zvtvz / zvt

modular quant framework.
https://zvt.readthedocs.io/en/latest/
MIT License
3.22k stars 860 forks source link

(sqlite3.IntegrityError) column id is not unique #117

Closed ranglang closed 4 years ago

ranglang commented 4 years ago

   timestamps=timestamps)
  File "/usr/local/lib/python3.7/site-packages/zvt-0.9.1-py3.7.egg/zvt/recorders/joinquant/quotes/jq_stock_kdata_recorder.py", line 143, in record
    df_to_db(df=df, data_schema=self.data_schema, provider=self.provider, force_update=self.force_update)
  File "/usr/local/lib/python3.7/site-packages/zvt-0.9.1-py3.7.egg/zvt/contract/api.py", line 454, in df_to_db
    df_current.to_sql(data_schema.__tablename__, db_engine, index=False, if_exists='append')
  File "/usr/local/lib/python3.7/site-packages/pandas/core/generic.py", line 2615, in to_sql
    method=method,
  File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 598, in to_sql
    method=method,
  File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 1406, in to_sql
    raise err
  File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 1398, in to_sql
    table.insert(chunksize, method=method)
  File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 830, in insert
    exec_insert(conn, keys, chunk_iter)
  File "/usr/local/lib/python3.7/site-packages/pandas/io/sql.py", line 747, in _execute_insert
    conn.execute(self.table.insert(), data)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1170, in _execute_context
    context)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 506, in do_executemany
    cursor.executemany(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) column id is not unique [SQL: 'INSERT INTO stock_1d_kdata (level, name, entity_id, close, timestamp, volume, code, provider, open, id, low, high, turnover) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-16 00:00:00.000000', 18536575, '002668', 'joinquant', 4.66, 'stock_sz_002668_2020-10-16', 4.61, 4.7, 86227186.86), ('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-19 00:00:00.000000', 0, '002668', 'joinquant', 4.7, 'stock_sz_002668_2020-10-19', 4.7, 4.7, 0.0), ('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-20 00:00:00.000000', 0, '002668', 'joinquant', 4.7, 'stock_sz_002668_2020-10-20', 4.7, 4.7, 0.0), ('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-21 00:00:00.000000', 0, '002668', 'joinquant', 4.7, 'stock_sz_002668_2020-10-21', 4.7, 4.7, 0.0), ('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-21 00:00:00.000000', 0, '002668', 'joinquant', 4.7, 'stock_sz_002668_2020-10-21', 4.7, 4.7, 0.0), ('1d', '奥马电器', 'stock_sz_002668', 4.7, '2020-10-23 00:00:00.000000', 0, '002668', 'joinquant', 4.7, 'stock_sz_002668_2020-10-23', 4.7, 4.7, 0.0), ('1d', '奥马电器', 'stock_sz_002668', 4.84, '2020-10-26 00:00:00.000000', 54935175, '002668', 'joinquant', 4.84, 'stock_sz_002668_2020-10-26', 4.79, 5.08, 269399105.34), ('1d', '奥马电器', 'stock_sz_002668', 4.68, '2020-10-27 00:00:00.000000', 28395532, '002668', 'joinquant', 4.77, 'stock_sz_002668_2020-10-27', 4.65, 4.84, 134329336.16), ('1d', '奥马电器', 'stock_sz_002668', 4.57, '2020-10-28 00:00:00.000000', 23822369, '002668', 'joinquant', 4.64, 'stock_sz_002668_2020-10-28', 4.51, 4.64, 108758590.8), ('1d', '奥马电器', 'stock_sz_002668', 4.5, '2020-10-29 00:00:00.000000', 20219276, '002668', 'joinquant', 4.49, 'stock_sz_002668_2020-10-29', 4.46, 4.53, 90971480.08))] (Background on this error at: http://sqlalche.me/e/gkpj)
2020-10-29 21:30:21,484  INFO  MainThread  count 4145
foolcage commented 4 years ago

df_to_db在最新版本里面已经加了 drop_duplicates参数,一般传入的df id已经去重,没必要设置。

jq kdata有时好像会取到重复数据,已经在recorder里面修复了。