search

springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 98 forks source link

Be able to pull files by specific day and time #7

Closed CloudandReact closed 7 years ago

CloudandReact commented 7 years ago

Hi,

I just wanted to know if it was possible for spark 1.6 to pull data by specific day. Like feb14 between 2am and 6am. Thanks

springml commented 7 years ago

You may need to rely on external tools to do scheduling e.g. databricks or zeppelin are two notebook servers that support scheduling of scripts written to run on spark. There may be other tools too.