Open sunayansaikia opened 5 years ago
@sunayansaikia - That was done to have a single file in SFTP
Hey @samuel-pt : is this a hard requirement? Can we not download multiple files for upload via SFTP? Can't this option be made configurable?
@sunayansaikia - Its not hard. we can just add a configurable parameter and use it. Following are needed
ok - cool. Will take a checkout and see.
@sunayansaikia, the below PR should be able to fix the coalesce numPartitions configuration https://github.com/springml/spark-sftp/pull/68
@shaikmanu797 : Great!
Hey folks,
Just wanted to understand why 'df.coalesce(1)' was done while writing the dataframe to DFS? Please refer code here: https://github.com/springml/spark-sftp/blob/master/src/main/scala/com/springml/spark/sftp/DefaultSource.scala#L249
Thanks