thombashi / sqlitebiter

A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
https://sqlitebiter.rtfd.io/
MIT License
850 stars 50 forks source link

sqlite3.OperationalError: duplicate column name #53

Closed chaffinched closed 6 years ago

chaffinched commented 6 years ago

On Ubuntu 16.04 (both a "real" install and in Bash in Ubuntu on Windows 10) I get the following error, which appears to be due to having column names with non-alphanumeric characters. In this case I have columns "ABCD>8.5" and "ABCD<8.5". I'm not sure if this is a problem with sqlitebiter, simplesqlite or something else?

sqlitebiter file complicated.xlsx Traceback (most recent call last): File "site-packages/simplesqlite/core.py", line 259, in execute_query sqlite3.OperationalError: duplicate column name: ABCD85

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "cli.py", line 12, in File "site-packages/click/core.py", line 722, in call File "site-packages/click/core.py", line 697, in main File "site-packages/click/core.py", line 1066, in invoke File "site-packages/click/core.py", line 895, in invoke File "site-packages/click/core.py", line 535, in invoke File "site-packages/click/decorators.py", line 17, in new_func File "sqlitebiter/sqlitebiter.py", line 286, in file File "sqlitebiter/_table_creator.py", line 22, in create File "site-packages/simplesqlite/core.py", line 1219, in create_table_from_tabledata File "site-packages/simplesqlite/core.py", line 1533, in __create_table_from_tabledata File "site-packages/simplesqlite/core.py", line 1114, in create_table File "site-packages/simplesqlite/core.py", line 270, in execute_query simplesqlite.error.OperationalError [144] Failed to execute script cli

thombashi commented 6 years ago

@chaffinched Thank you for your report. I had fix the problem at the sqlitebiter 0.15.1. Could you try with this version?

The problem is non-alphanumeric characters are removed while the SQLite conversion. So, ABCD>8.5 and ABCD<8.5 are considered a duplicate column as ABCD85. Now, sqlitebiter accepts non-alphanumeric characters.

chaffinched commented 6 years ago

Thanks, @thombashi that works perfectly for the 0.15.1 .deb file.

Unfortunately, the problem still exists in the Windows version, now that #52 is fixed.

sqlitebiter.exe file Book1.xlsx [INFO] sqlitebiter file: convert 'Book1.xlsx' to 'Sheet1' table Traceback (most recent call last): File "site-packages\simplesqlite\core.py", line 259, in execute_query sqlite3.OperationalError: duplicate column name: ABCD85 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cli.py", line 15, in File "site-packages\click\core.py", line 722, in call File "site-packages\click\core.py", line 697, in main File "site-packages\click\core.py", line 1066, in invoke File "site-packages\click\core.py", line 895, in invoke File "site-packages\click\core.py", line 535, in invoke File "site-packages\click\decorators.py", line 17, in new_func File "sqlitebiter\sqlitebiter.py", line 230, in file File "sqlitebiter_table_creator.py", line 22, in create File "site-packages\simplesqlite\core.py", line 1185, in create_table_from_tabledata File "site-packages\simplesqlite\core.py", line 1487, in __create_table_from_tabledata File "site-packages\simplesqlite\core.py", line 1093, in create_table File "site-packages\simplesqlite\core.py", line 270, in execute_query simplesqlite.error.OperationalError [13116] Failed to execute script cli

thombashi commented 6 years ago

@chaffinched Sorry to bother you. I had rebuild the Windows executable binary. I believe this binary fixed the issue.

sha256 hash: cd1e05148360c7f357e5c8be5dec01f6ad2fc34ab4fa639a2541c6494866ea69

chaffinched commented 6 years ago

@thombashi Great, this works perfectly now! Thanks for all of your help.