d6t / d6tstack

Quickly ingest messy CSV and XLS files. Export to clean pandas, SQL, parquet
MIT License
195 stars 46 forks source link

Encoding error reading csv files to combine #28

Open lucky-luk3 opened 3 years ago

lucky-luk3 commented 3 years ago

I have csv files and I'm trying to fix different columns between them.

The function d6tstack.combine_csv.CombinerCSV works fine in other cases but now I have problems with the encoding.

I need to pass the parameter encoding="latin-1" to Pandas read_csv but it doesn't work. In the documentation I found that is possible to pass read_csv_params={"encoding" : "latin-1"} but it doesn't work, it doesn't apply this encoding.

I tried reading the same file directly with Pandas and whit the parameter encoding works fine.

Are there another posivility to resolve it? Thanks in advance.