File download problem - Githubissues

springml / spark-sftp

Spark connector for SFTP

Apache License 2.0

100 stars 98 forks source link

File download problem #29

Closed luozhao528 closed 6 years ago

luozhao528 commented 6 years ago

Dataset df = sqlContext.read(). format("com.springml.spark.sftp"). option("host", ""). option("username", ""). option("password", ""). option("fileType", ""). load("/sparktest/sparkfile0.bcp"); spark have two worker, worker1 downloaded(temp/111.josn),but worker2 not download(temp/null),why?

samuel-pt commented 6 years ago

This connector does not split the job into multiple workers as the file from SFTP server is not copied in parallel and hence you are seeing the file in only one worker