samuel-pt
commented
6 years ago @bagopalan - This make sense. But I think these configurations should be added in SparkConf itself.
Which means, when user provides password in sc.read|sc.write, specified password will be used. Else, it will look into these properties.
And what does the user strategy will do? I can understand file and batch but not the user.
When we read or write from/into SFTP servers, we need to pass the passwords as options. We have our PayPal notebooks which are customized on top of Jupiter notebooks from which we need to invoke the read and write API in which case we have to put the password and when we check in the code in Github it is not secured.
Should we create an option to put the password in a local/hdfs file.
sftp.file.password.strategy -> can be user or file or batch ( the batch option is for password less SSH) sftp.file.password.path -> get the path (It could be local or hdfs path) sftp.file.password.source -> can be local or hdfs.
Once we get the options (above) , we can write use this to have a module to read password files and use that instead of directly getting the password itself as option.
We have implemented this in our Paypals code base (https://github.com/paypal/gimel) and I can implement the same in Spark-SFTP as well.
@samuel-pt , Please suggest whether these changes make sense.