Avoid FileSystem.get(Configuration)

OryxProject / oryx

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

http://oryx.io

Apache License 2.0

1.79k stars 405 forks source link

Avoid FileSystem.get(Configuration) #329

Closed srowen closed 7 years ago

srowen commented 7 years ago

SaveToHDFSFunction has an instance where a Hadoop FileSystem is obtained not for a path but according to the default Hadoop config. This is a problem when reading from S3, for example. It should simply respect the scheme from the path.