When writing dataframes with this library, I found that the file name written on the FTP server is the part file name used when writing the local temp file.
I found that the issue is happening in ChannelSftp.java, around line 442 where "_dst" is set. Rather than using the given target, it winds up adding the part file name on to the destination.
So what gets written is /folder/some_file.csv/part-0000-......xyz.csv
When writing dataframes with this library, I found that the file name written on the FTP server is the part file name used when writing the local temp file.
df.write .format("com.springml.spark.sftp") .option("host", host) .option("port", port) .option("username", userName) .option("password", password) .option("fileType", "csv") .save("/folder/some_file.csv")I found that the issue is happening in ChannelSftp.java, around line 442 where "_dst" is set. Rather than using the given target, it winds up adding the part file name on to the destination.
So what gets written is /folder/some_file.csv/part-0000-......xyz.csv