simonw / csvs-to-sqlite

Convert CSV files into a SQLite database
Apache License 2.0
875 stars 69 forks source link

When input CSV has a column named 'table_name', an error is raised: "sqlite3.InterfaceError: Error binding parameter 0" #70

Open dannguyen opened 4 years ago

dannguyen commented 4 years ago

(This is for csvs-to-sqlite=1.0, using Python 3.7)

How to reproduce the error

Given a CSV that has table_name as one of its column headers, e.g. this OSHA data dictionary.csv

mkdir -p /tmp/csvtest && cd /tmp/csvtest
curl -LO https://raw.githubusercontent.com/storydrivendatasets/osha-enforcement-catalog/master/data/collected/osha/stash/osha_data_dictionary/osha_data_dictionary.csv

csvs-to-sqlite osha_data_dictionary.csv testdb.sqlite

csvs-to-sqlite throws an error:

  File "../python3.7/site-packages/csvs_to_sqlite/cli.py", line 198, in cli
    if replace_tables and table_exists(conn, df.table_name):
  File "../python3.7/site-packages/csvs_to_sqlite/utils.py", line 260, in table_exists
    [table],
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.

How the error is resolved

If the input CSV's table_name header is changed to anything else, e.g. tbl_name, then csvs-to-sqlite works as expected

Suggested fix

Haven't looked at the code yet, but my initial thought is that it'd be nice if csvs-to-sqlite silently handled this somehow, maybe changing table_name (and any other reserved words), to a string with a specified prefix/suffix, e.g. table_name__fixed__. Then again, maybe silently handling this kind of thing can lead to messy usecases later?

fgregg commented 2 years ago

i just ran into the same problem on the same data