search

simonw / csvs-to-sqlite

Convert CSV files into a SQLite database
Apache License 2.0
875 stars 69 forks source link

Add option to pass na_filter into pd.read_csv() - Dealing with CSV containing NA as string #44

Open dazzag24 opened 5 years ago

dazzag24 commented 5 years ago

I have a CSV detailing info about airports. The country code column is encoded using 2 character ISO-3166 codes. https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes

It turns out that the code for Namibia is "NA" and this results in the values being stripped as it converts from CSV to SQLite DB.

I was able to solve this by adding na_filter=False to the pd.read_csv() call in utils.py https://github.com/simonw/csvs-to-sqlite/blob/dccbf65b37bc9eed50e9edb80a42f257e93edb1f/csvs_to_sqlite/utils.py#L24

Would you consider adding an option to allow this flag to be passed into the pd.read_csv call?

Thanks Darren

frafra commented 2 years ago

I agree, I have the same issue.