Closed aarafeh2017 closed 6 years ago
What does the crc.csv file contain? Does it contain the values present in Dataframe? Also please send us the code snippet of you job
@samuel-pt, it is csv.crc file, it is always generated with any csv file, it is binary file, it seems for file validation, i hope if there is a way to suppress that file, but it is always generated and always the one that is uploaded to FTP instead of the csv file, here is sample of my code attached, i just try to convert data coming from oracle to csv and upload to FTP. Code.txt
@aarafeh2017 - I've pushed a fix to this issue. Could you please pull the source, build and run your code with the built spark-package?
@samuel-pt , Thank you! it is working fine now.
When i try to write a dataframe to FTP as CSV, the crc.csv file is the one that is uploaded to FTP!
Here is the console output sample : 18/02/11 11:32:36 INFO DefaultSource: Copying C:\Users\aarafeh\AppData\Local\Temp\spark_sftp_connection_temp778.part-00000-6c9f54e2-34b2-42f1-bc91-d99a70a9fcb4-c000.csv.crc to /J28446_Engage/Test/testfile.csv 18/02/11 11:32:38 INFO SFTPClient: Copying files from C:\Users\aarafeh\AppData\Local\Temp\spark_sftp_connection_temp778.part-00000-6c9f54e2-34b2-42f1-bc91-d99a70a9fcb4-c000.csv.crc to /J28446_Engage/Test/testfile.csv 18/02/11 11:32:39 INFO SFTPClient: Copied files successfully...