Open rsrini7 opened 5 years ago
Csvdiff is meant for datasets that fit into memory. It's not a streaming comparison. How big are the files you're trying to compare?
ok thanks. I am comparing around 80 MB files. Data from : https://blog.majestic.com/development/majestic-million-csv-daily/ http://downloads.majestic.com/majestic_million.csv
Large File Error:
File "i:\anaconda3\lib\site-packages\csvdiff\records.py", line 53, in
for r in record_seq
MemoryError
Unicode Decode Error:
return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5148: character maps to