search

springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 99 forks source link

CRC file uploaded to FTP instead of CSV #18

Closed aarafeh2017 closed 6 years ago

aarafeh2017 commented 6 years ago

When i try to write a dataframe to FTP as CSV, the crc.csv file is the one that is uploaded to FTP!

Here is the console output sample : 18/02/11 11:32:36 INFO DefaultSource: Copying C:\Users\aarafeh\AppData\Local\Temp\spark_sftp_connection_temp778.part-00000-6c9f54e2-34b2-42f1-bc91-d99a70a9fcb4-c000.csv.crc to /J28446_Engage/Test/testfile.csv 18/02/11 11:32:38 INFO SFTPClient: Copying files from C:\Users\aarafeh\AppData\Local\Temp\spark_sftp_connection_temp778.part-00000-6c9f54e2-34b2-42f1-bc91-d99a70a9fcb4-c000.csv.crc to /J28446_Engage/Test/testfile.csv 18/02/11 11:32:39 INFO SFTPClient: Copied files successfully...

samuel-pt commented 6 years ago

What does the crc.csv file contain? Does it contain the values present in Dataframe? Also please send us the code snippet of you job

aarafeh2017 commented 6 years ago

@samuel-pt, it is csv.crc file, it is always generated with any csv file, it is binary file, it seems for file validation, i hope if there is a way to suppress that file, but it is always generated and always the one that is uploaded to FTP instead of the csv file, here is sample of my code attached, i just try to convert data coming from oracle to csv and upload to FTP. Code.txt

samuel-pt commented 6 years ago

@aarafeh2017 - I've pushed a fix to this issue. Could you please pull the source, build and run your code with the built spark-package?

aarafeh2017 commented 6 years ago

@samuel-pt , Thank you! it is working fine now.