[FEATURE REQUEST]: Spark version 3.1.3 is not supported by current dotnet on spark code. This is preventing Migration to HDI 5.0 which uses spark version 3.1.3

dotnet / spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

https://dot.net/spark

MIT License

2.02k stars 312 forks source link

Open aadithya0701 opened 1 year ago

aadithya0701 commented 1 year ago

Is your feature request related to a problem? Please describe.

Spark version 3.1.3 is not supported by current dotnet on spark code. This is preventing Migration to HDI 5.0 which uses spark version 3.1.3

Describe the solution you'd like Adding 3.1.3 as supported verion in dotnetrunner class will help resolve this issue.

Describe alternatives you've considered Nil

Additional context Nil

dbeavon commented 5 months ago

Hi @aadithya0701 I've tested this change. It is an easy one to make. I'm on my way into HDI from Synapse as well.

We just need a committer on the project. Do you happen to know anyone that can help us commit PR's? If so I can create the PR for you.