sensiblecodeio / scraperwiki-python

ScraperWiki Python library for scraping and saving data
https://scraperwiki.com
BSD 2-Clause "Simplified" License
160 stars 69 forks source link

Unicode column names fail the second time you use them. #86

Closed scraperdragon closed 8 years ago

scraperdragon commented 8 years ago
>>> import scraperwiki
>>> scraperwiki.sqlite.save(data = {"i": 1, u"a\xa0b": 1}, unique_keys = ['i'])
>>> scraperwiki.sqlite.save(data = {"i": 1, u"a\xa0b": 1}, unique_keys = ['i'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dragon/.local/lib/python2.7/site-packages/scraperwiki/sql.py", line 202, in save
    fit_row(connection, row, unique_keys)
  File "/home/dragon/.local/lib/python2.7/site-packages/scraperwiki/sql.py", line 354, in fit_row
    add_column(connection, new_column)
  File "/home/dragon/.local/lib/python2.7/site-packages/scraperwiki/sql.py", line 374, in add_column
    connection.execute(stmt)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 69, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 895, in _execute_ddl
    compiled
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1070, in _execute_context
    context)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception
    exc_info
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "/home/dragon/.local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (OperationalError) duplicate column name: a b u'ALTER TABLE swdata ADD COLUMN "a\xa0b" BIGINT' ()

Caused, I suspect, by sql.py:345 -- if not str(new_column) in list(_State.table.columns.keys()) which doesn't preserve unicodeness and hence breaks.

`