search

springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 98 forks source link

Downloading Multiple Files into the dataframe #34

Closed viveknair89 closed 5 years ago

viveknair89 commented 6 years ago

val df = spark.read. format("com.springml.spark.sftp"). option("username", "username"). option("pem", "location"). option("host", "hostname"). option("port","portname"). option("fileType", "filetype"). option("codec","codec"). option("inferSchema", "true"). option("hdfsTempLocation", "/tmp/"). *load("/dir/file__")**

Basically, I want to specify glob patterns for filenames and fetch multiple files from the sftp server. Is this currently possible using springml?

Note: I got a "Copying multiple files, but destination is missing or a file." error while I tried executing the above code

samuel-pt commented 5 years ago

@viveknair89 - Sorry, currently this is not supported

nicknobad commented 5 years ago

would this feature (asked by @viveknair89) be supported in the future?