Azure Data Lake Store Support

indy-3rdman / docker-dotnet-spark

A .NET for Apache Spark docker image (3rdman/dotnet-spark)

https://hub.docker.com/r/3rdman/dotnet-spark

Apache License 2.0

19 stars 9 forks source link

Azure Data Lake Store Support #18

Open strtdusty opened 2 years ago

strtdusty commented 2 years ago

Would you be interested in a PR adding support for communicating with Azure Datalake?

The changes I have made to enable it locally:

Switch the spark package to use "without-hadoop"
Download the full hadoop dist.
Upgrade Hadoop version to 3.2.2
Add a couple of environment variables for HADOOP
Add the necessary hadoop/azure jar files to $SPARK_HOME/jars

I am specifically interested in having it in the runtime image but could see its use in the interactive as well.

strtdusty commented 2 years ago

Minimally, how about at least a PR to rev the version of HADOOP to something greater than 2.7 (3.2 would be the best I think)? I think I could handle the remainder with spark-submit package parameters.

indy-3rdman commented 2 years ago

@strtdusty, I've added a new image type runtime-hadoop that comes with a full installation of hadoop. You should be able to try it out via docker pull 3rdman/dotnet-spark:2.1.1-3.2.1-3.2.3-hadoop for example.

Could you please give it a try and let me know whether it works as expected?