matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 276 forks source link

[Feature Request]: Problem with sqlalchemy #2174

Closed dengn closed 2 years ago

dengn commented 2 years ago

Is there an existing issue for the same bug?

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):05e8fdd15a00263b7808d478458ad7c263e0a5a6
- Hardware parameters: Macbook pro m1, arm chipset, 16GB memory
- OS type: OSX
- Others:

Actual Behavior

I wrote some python code to connect with some public dataset. I used sqlalchemy as the database connector. It works well with MySQL, but not with MatrixOne. It look like sqlalchemy cannot automatically create table with MatrixOne. Below is the error log of my python code.

Create connection complete
2022-04-07 17:34:16,377 INFO sqlalchemy.engine.Engine SHOW VARIABLES LIKE 'sql_mode'
2022-04-07 17:34:16,377 - sqlalchemy.engine.Engine - INFO - SHOW VARIABLES LIKE 'sql_mode'
2022-04-07 17:34:16,378 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:34:16,378 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
/Users/nandeng/Test/tushare/tushare_test_matrixone.py:13: SAWarning: Could not retrieve SQL_MODE; please ensure the MySQL user has permissions to SHOW VARIABLES
  engine.execute("use " + database_name + ";")
2022-04-07 17:34:16,379 INFO sqlalchemy.engine.Engine SHOW VARIABLES LIKE 'lower_case_table_names'
2022-04-07 17:34:16,379 - sqlalchemy.engine.Engine - INFO - SHOW VARIABLES LIKE 'lower_case_table_names'
2022-04-07 17:34:16,379 INFO sqlalchemy.engine.Engine [generated in 0.00008s] {}
2022-04-07 17:34:16,379 - sqlalchemy.engine.Engine - INFO - [generated in 0.00008s] {}
Traceback (most recent call last):
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect
    dialect.initialize(c)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2961, in initialize
    default.DefaultDialect.initialize(self, connection)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 394, in initialize
    self.server_version_info = self._get_server_version_info(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/base.py", line 2750, in _get_server_version_info
    cursor.execute("SELECT VERSION()")
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1105, 'unknown error:[42000]table not exist')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
    return fn()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
    return _ConnectionFairy._checkout(self)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
    rec = pool._do_get()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
    self.__connect()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 622, in __connect
    pool.dispatch.connect.for_modify(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 329, in _exec_w_sync_on_first_run
    self(*args, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 343, in __call__
    fn(*args, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1691, in go
    return once_fn(*arg, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 682, in first_connect
    dialect.do_rollback(c.connection)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 667, in do_rollback
    dbapi_connection.rollback()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 480, in rollback
    self._read_ok_packet()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 443, in _read_ok_packet
    pkt = self._read_packet()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1105, "unknown error:[03000]unexpected statement: 'rollback'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/nandeng/Test/tushare/tushare_test_matrixone.py", line 13, in <module>
    engine.execute("use " + database_name + ";")
  File "<string>", line 2, in execute
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 390, in warned
    return fn(*args, **kwargs)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3107, in execute
    connection = self.connect(close_with_result=True)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
    return fn()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
    return _ConnectionFairy._checkout(self)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
    rec = pool._do_get()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
    self.__connect()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 622, in __connect
    pool.dispatch.connect.for_modify(
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 329, in _exec_w_sync_on_first_run
    self(*args, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 343, in __call__
    fn(*args, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1691, in go
    return once_fn(*arg, **kw)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 682, in first_connect
    dialect.do_rollback(c.connection)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 667, in do_rollback
    dbapi_connection.rollback()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 480, in rollback
    self._read_ok_packet()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 443, in _read_ok_packet
    pkt = self._read_packet()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/Users/nandeng/opt/anaconda3/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1105, "unknown error:[03000]unexpected statement: 'rollback'")
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Expected Behavior

MySQL log with this python code is


Create connection complete
2022-04-07 17:32:20,869 INFO sqlalchemy.engine.Engine SHOW VARIABLES LIKE 'sql_mode'
2022-04-07 17:32:20,869 - sqlalchemy.engine.Engine - INFO - SHOW VARIABLES LIKE 'sql_mode'
2022-04-07 17:32:20,869 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:32:20,869 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
2022-04-07 17:32:20,880 INFO sqlalchemy.engine.Engine SHOW VARIABLES LIKE 'lower_case_table_names'
2022-04-07 17:32:20,880 - sqlalchemy.engine.Engine - INFO - SHOW VARIABLES LIKE 'lower_case_table_names'
2022-04-07 17:32:20,880 INFO sqlalchemy.engine.Engine [generated in 0.00013s] {}
2022-04-07 17:32:20,880 - sqlalchemy.engine.Engine - INFO - [generated in 0.00013s] {}
2022-04-07 17:32:20,883 INFO sqlalchemy.engine.Engine SELECT DATABASE()
2022-04-07 17:32:20,883 - sqlalchemy.engine.Engine - INFO - SELECT DATABASE()
2022-04-07 17:32:20,883 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:32:20,883 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
2022-04-07 17:32:20,885 INFO sqlalchemy.engine.Engine use tushare_test;
2022-04-07 17:32:20,885 - sqlalchemy.engine.Engine - INFO - use tushare_test;
2022-04-07 17:32:20,885 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:32:20,885 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
Enter database successfully
2022-04-07 17:32:22,407 INFO sqlalchemy.engine.Engine SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = %(table_schema)s AND table_name = %(table_name)s
2022-04-07 17:32:22,407 - sqlalchemy.engine.Engine - INFO - SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = %(table_schema)s AND table_name = %(table_name)s
2022-04-07 17:32:22,408 INFO sqlalchemy.engine.Engine [generated in 0.00032s] {'table_schema': 'tushare_test', 'table_name': 'df1'}
2022-04-07 17:32:22,408 - sqlalchemy.engine.Engine - INFO - [generated in 0.00032s] {'table_schema': 'tushare_test', 'table_name': 'df1'}
2022-04-07 17:32:22,415 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2022-04-07 17:32:22,415 - sqlalchemy.engine.Engine - INFO - BEGIN (implicit)
2022-04-07 17:32:22,416 INFO sqlalchemy.engine.Engine
CREATE TABLE tushare_test.df1 (
    `index` BIGINT,
    ts_code TEXT,
    trade_date TEXT,
    open FLOAT(53),
    high FLOAT(53),
    low FLOAT(53),
    close FLOAT(53),
    pre_close FLOAT(53),
    `change` FLOAT(53),
    pct_chg FLOAT(53),
    vol FLOAT(53),
    amount FLOAT(53)
)

2022-04-07 17:32:22,416 - sqlalchemy.engine.Engine - INFO -
CREATE TABLE tushare_test.df1 (
    `index` BIGINT,
    ts_code TEXT,
    trade_date TEXT,
    open FLOAT(53),
    high FLOAT(53),
    low FLOAT(53),
    close FLOAT(53),
    pre_close FLOAT(53),
    `change` FLOAT(53),
    pct_chg FLOAT(53),
    vol FLOAT(53),
    amount FLOAT(53)
)

2022-04-07 17:32:22,416 INFO sqlalchemy.engine.Engine [no key 0.00013s] {}
2022-04-07 17:32:22,416 - sqlalchemy.engine.Engine - INFO - [no key 0.00013s] {}
2022-04-07 17:32:22,459 INFO sqlalchemy.engine.Engine CREATE INDEX ix_tushare_test_df1_index ON tushare_test.df1 (`index`)
2022-04-07 17:32:22,459 - sqlalchemy.engine.Engine - INFO - CREATE INDEX ix_tushare_test_df1_index ON tushare_test.df1 (`index`)
2022-04-07 17:32:22,459 INFO sqlalchemy.engine.Engine [no key 0.00013s] {}
2022-04-07 17:32:22,459 - sqlalchemy.engine.Engine - INFO - [no key 0.00013s] {}
2022-04-07 17:32:22,487 INFO sqlalchemy.engine.Engine COMMIT
2022-04-07 17:32:22,487 - sqlalchemy.engine.Engine - INFO - COMMIT
2022-04-07 17:32:22,494 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2022-04-07 17:32:22,494 - sqlalchemy.engine.Engine - INFO - BEGIN (implicit)
2022-04-07 17:32:22,549 INFO sqlalchemy.engine.Engine INSERT INTO tushare_test.df1 (`index`, ts_code, trade_date, open, high, low, close, pre_close, `change`, pct_chg, vol, amount) VALUES (%(index)s, %(ts_code)s, %(trade_date)s, %(open)s, %(high)s, %(low)s, %(close)s, %(pre_close)s, %(change)s, %(pct_chg)s, %(vol)s, %(amount)s)
2022-04-07 17:32:22,549 - sqlalchemy.engine.Engine - INFO - INSERT INTO tushare_test.df1 (`index`, ts_code, trade_date, open, high, low, close, pre_close, `change`, pct_chg, vol, amount) VALUES (%(index)s, %(ts_code)s, %(trade_date)s, %(open)s, %(high)s, %(low)s, %(close)s, %(pre_close)s, %(change)s, %(pct_chg)s, %(vol)s, %(amount)s)
2022-04-07 17:32:22,549 INFO sqlalchemy.engine.Engine [generated in 0.04584s] ({'index': 0, 'ts_code': '001914.SZ', 'trade_date': '20180702', 'open': 7.18, 'high': 7.22, 'low': 6.98, 'close': 7.06, 'pre_close': 7.09, 'change': -0.03, 'pct_chg': -0.42, 'vol': 41067.34, 'amount': 29080.376}, {'index': 1, 'ts_code': '603999.SH', 'trade_date': '20180702', 'open': 5.94, 'high': 5.95, 'low': 5.77, 'close': 5.8, 'pre_close': 5.96, 'change': -0.16, 'pct_chg': -2.68, 'vol': 34033.2, 'amount': 19956.189}, {'index': 2, 'ts_code': '603998.SH', 'trade_date': '20180702', 'open': 8.65, 'high': 8.65, 'low': 8.33, 'close': 8.4, 'pre_close': 8.55, 'change': -0.15, 'pct_chg': -1.75, 'vol': 16143.56, 'amount': 13724.01}, {'index': 3, 'ts_code': '603996.SH', 'trade_date': '20180702', 'open': 9.57, 'high': 10.68, 'low': 9.51, 'close': 10.68, 'pre_close': 9.71, 'change': 0.97, 'pct_chg': 9.99, 'vol': 246789.82, 'amount': 252657.161}, {'index': 4, 'ts_code': '603993.SH', 'trade_date': '20180702', 'open': 6.26, 'high': 6.27, 'low': 6.08, 'close': 6.14, 'pre_close': 6.29, 'change': -0.15, 'pct_chg': -2.38, 'vol': 699425.2, 'amount': 430950.44}, {'index': 5, 'ts_code': '603991.SH', 'trade_date': '20180702', 'open': 22.62, 'high': 23.3, 'low': 22.58, 'close': 22.71, 'pre_close': 22.82, 'change': -0.11, 'pct_chg': -0.48, 'vol': 2967.0, 'amount': 6803.869}, {'index': 6, 'ts_code': '603990.SH', 'trade_date': '20180702', 'open': 33.37, 'high': 34.56, 'low': 32.61, 'close': 33.09, 'pre_close': 33.94, 'change': -0.85, 'pct_chg': -2.5, 'vol': 3175.0, 'amount': 10616.508}, {'index': 7, 'ts_code': '603989.SH', 'trade_date': '20180702', 'open': 22.45, 'high': 22.45, 'low': 21.6, 'close': 21.91, 'pre_close': 22.47, 'change': -0.56, 'pct_chg': -2.49, 'vol': 12437.29, 'amount': 27288.538}  ... displaying 10 of 3335 total bound parameter sets ...  {'index': 3333, 'ts_code': '000002.SZ', 'trade_date': '20180702', 'open': 24.5, 'high': 24.55, 'low': 22.52, 'close': 22.8, 'pre_close': 24.6, 'change': -1.8, 'pct_chg': -7.32, 'vol': 846203.86, 'amount': 1981131.638}, {'index': 3334, 'ts_code': '000001.SZ', 'trade_date': '20180702', 'open': 9.05, 'high': 9.05, 'low': 8.55, 'close': 8.61, 'pre_close': 9.09, 'change': -0.48, 'pct_chg': -5.28, 'vol': 1315520.13, 'amount': 1158545.868})
2022-04-07 17:32:22,549 - sqlalchemy.engine.Engine - INFO - [generated in 0.04584s] ({'index': 0, 'ts_code': '001914.SZ', 'trade_date': '20180702', 'open': 7.18, 'high': 7.22, 'low': 6.98, 'close': 7.06, 'pre_close': 7.09, 'change': -0.03, 'pct_chg': -0.42, 'vol': 41067.34, 'amount': 29080.376}, {'index': 1, 'ts_code': '603999.SH', 'trade_date': '20180702', 'open': 5.94, 'high': 5.95, 'low': 5.77, 'close': 5.8, 'pre_close': 5.96, 'change': -0.16, 'pct_chg': -2.68, 'vol': 34033.2, 'amount': 19956.189}, {'index': 2, 'ts_code': '603998.SH', 'trade_date': '20180702', 'open': 8.65, 'high': 8.65, 'low': 8.33, 'close': 8.4, 'pre_close': 8.55, 'change': -0.15, 'pct_chg': -1.75, 'vol': 16143.56, 'amount': 13724.01}, {'index': 3, 'ts_code': '603996.SH', 'trade_date': '20180702', 'open': 9.57, 'high': 10.68, 'low': 9.51, 'close': 10.68, 'pre_close': 9.71, 'change': 0.97, 'pct_chg': 9.99, 'vol': 246789.82, 'amount': 252657.161}, {'index': 4, 'ts_code': '603993.SH', 'trade_date': '20180702', 'open': 6.26, 'high': 6.27, 'low': 6.08, 'close': 6.14, 'pre_close': 6.29, 'change': -0.15, 'pct_chg': -2.38, 'vol': 699425.2, 'amount': 430950.44}, {'index': 5, 'ts_code': '603991.SH', 'trade_date': '20180702', 'open': 22.62, 'high': 23.3, 'low': 22.58, 'close': 22.71, 'pre_close': 22.82, 'change': -0.11, 'pct_chg': -0.48, 'vol': 2967.0, 'amount': 6803.869}, {'index': 6, 'ts_code': '603990.SH', 'trade_date': '20180702', 'open': 33.37, 'high': 34.56, 'low': 32.61, 'close': 33.09, 'pre_close': 33.94, 'change': -0.85, 'pct_chg': -2.5, 'vol': 3175.0, 'amount': 10616.508}, {'index': 7, 'ts_code': '603989.SH', 'trade_date': '20180702', 'open': 22.45, 'high': 22.45, 'low': 21.6, 'close': 21.91, 'pre_close': 22.47, 'change': -0.56, 'pct_chg': -2.49, 'vol': 12437.29, 'amount': 27288.538}  ... displaying 10 of 3335 total bound parameter sets ...  {'index': 3333, 'ts_code': '000002.SZ', 'trade_date': '20180702', 'open': 24.5, 'high': 24.55, 'low': 22.52, 'close': 22.8, 'pre_close': 24.6, 'change': -1.8, 'pct_chg': -7.32, 'vol': 846203.86, 'amount': 1981131.638}, {'index': 3334, 'ts_code': '000001.SZ', 'trade_date': '20180702', 'open': 9.05, 'high': 9.05, 'low': 8.55, 'close': 8.61, 'pre_close': 9.09, 'change': -0.48, 'pct_chg': -5.28, 'vol': 1315520.13, 'amount': 1158545.868})
2022-04-07 17:32:22,681 INFO sqlalchemy.engine.Engine COMMIT
2022-04-07 17:32:22,681 - sqlalchemy.engine.Engine - INFO - COMMIT
2022-04-07 17:32:22,683 INFO sqlalchemy.engine.Engine SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = %(table_schema)s AND table_name = %(table_name)s
2022-04-07 17:32:22,683 - sqlalchemy.engine.Engine - INFO - SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = %(table_schema)s AND table_name = %(table_name)s
2022-04-07 17:32:22,683 INFO sqlalchemy.engine.Engine [cached since 0.2761s ago] {'table_schema': 'tushare_test', 'table_name': 'SELECT * FROM df1'}
2022-04-07 17:32:22,683 - sqlalchemy.engine.Engine - INFO - [cached since 0.2761s ago] {'table_schema': 'tushare_test', 'table_name': 'SELECT * FROM df1'}
2022-04-07 17:32:22,685 INFO sqlalchemy.engine.Engine SELECT * FROM df1
2022-04-07 17:32:22,685 - sqlalchemy.engine.Engine - INFO - SELECT * FROM df1
2022-04-07 17:32:22,685 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:32:22,685 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
      index    ts_code trade_date   open   high    low  close  pre_close  change  pct_chg         vol       amount
0         0  001914.SZ   20180702   7.18   7.22   6.98   7.06       7.09   -0.03    -0.42    41067.34    29080.376
1         1  603999.SH   20180702   5.94   5.95   5.77   5.80       5.96   -0.16    -2.68    34033.20    19956.189
2         2  603998.SH   20180702   8.65   8.65   8.33   8.40       8.55   -0.15    -1.75    16143.56    13724.010
3         3  603996.SH   20180702   9.57  10.68   9.51  10.68       9.71    0.97     9.99   246789.82   252657.161
4         4  603993.SH   20180702   6.26   6.27   6.08   6.14       6.29   -0.15    -2.38   699425.20   430950.440
...     ...        ...        ...    ...    ...    ...    ...        ...     ...      ...         ...          ...
3330   3330  000006.SZ   20180702   5.83   5.87   5.43   5.50       5.86   -0.36    -6.14   153168.58    86251.733
3331   3331  000005.SZ   20180702   2.97   2.97   2.89   2.90       2.96   -0.06    -2.03    42485.00    12467.476
3332   3332  000004.SZ   20180702  19.58  19.58  18.90  19.10      19.60   -0.50    -2.55     2210.20     4248.495
3333   3333  000002.SZ   20180702  24.50  24.55  22.52  22.80      24.60   -1.80    -7.32   846203.86  1981131.638
3334   3334  000001.SZ   20180702   9.05   9.05   8.55   8.61       9.09   -0.48    -5.28  1315520.13  1158545.868

[3335 rows x 12 columns]
2022-04-07 17:32:22,778 INFO sqlalchemy.engine.Engine drop table df1
2022-04-07 17:32:22,778 - sqlalchemy.engine.Engine - INFO - drop table df1
2022-04-07 17:32:22,778 INFO sqlalchemy.engine.Engine [raw sql] {}
2022-04-07 17:32:22,778 - sqlalchemy.engine.Engine - INFO - [raw sql] {}
2022-04-07 17:32:22,780 INFO sqlalchemy.engine.Engine COMMIT
2022-04-07 17:32:22,780 - sqlalchemy.engine.Engine - INFO - COMMIT

Steps to Reproduce

I'm accessing a public dataset to insert some data into a database like MatrixOne.
Below is my python code:

import sqlalchemy
from sqlalchemy import create_engine
import tushare as ts
import pandas as pd

ts.set_token('XXXXXXXXXXXXXX')
pro = ts.pro_api()

engine = create_engine("mysql+pymysql://root:XXXXXX@localhost:3306/tushare_test?charset=utf8", echo=True)
print('Create connection complete')
database_name = "tushare_test";
engine.execute("use " + database_name + ";")
print('Enter database successfully')

df1 = pro.daily(start_date='20180701', end_date='20180702')
pd.io.sql.to_sql(df1, 'df1', engine, schema=database_name, if_exists='replace')

sql_cmd = "SELECT * FROM df1"
df_out = pd.read_sql(sql=sql_cmd, con=engine)
print(df_out)


### Additional information

_No response_
daviszhen commented 2 years ago

it is similar to the jdbc compatibility

dengn commented 2 years ago

sqlalchemy support is added as 0.6 feature.

sukki37 commented 2 years ago

@dengn can you make sure current priority on this?

badboynt1 commented 2 years ago

sqlalchemy can be supported now. test script:

import sqlalchemy from sqlalchemy import exists, Column, Integer, String, ForeignKey, DateTime, Text, func from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker engine = create_engine("mysql+pymysql://dump:111@127.0.0.1:6001/motest?charset=utf8", echo=True) print('Create connection complete') database_name = "motest"; engine.execute("use " + database_name + ";") print('Enter database successfully') engine.execute("drop table if exists t1 ;") engine.execute("create table t1(c1 int, c2 int) ;") print('Create Table successfully') DBSession = sessionmaker(bind=engine) session = DBSession() session.execute('insert into t1 values(1,1);')
session.execute('insert into t1 values(2,2);')
session.execute('insert into t1 values(3,3);')
print('insert successfully') res = session.execute('select from t1')
all_res_list = res.fetchall()
print(all_res_list) session.execute('delete from t1 where c1=1;')
print('delete successfully') res = session.execute('select
from t1')
all_res_list = res.fetchall()
print(all_res_list) session.execute('update t1 set c2=10 where c1=2;')
print('update successfully') res = session.execute('select * from t1')
all_res_list = res.fetchall()
print(all_res_list)

badboynt1 commented 2 years ago

can not support create index statement and collation now. errors like:

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (20105, "not supported: statement: 'create index ix_tushare_test_df1_index on tushare_test.df1 (index)'")

/root/test/python/Flask-SQLAlchemy-RESTful-CRUD/.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py:383: SAWarning: Exception attempting to detect unicode returns: ProgrammingError('(MySQLdb._exceptions.ProgrammingError) (1064, \'SQL parser error: You have an error in your SQL syntax; check the manual that corresponds to your MatrixOne server version for the right syntax to use. syntax error at line 1 column 53 near " CHARACTER SET utf8mb4) COLLATE utf8mb4_bin AS anon_1";\')') util.warn(

dengn commented 2 years ago

The demo code doesn't get passed:

nandeng-macbookpro:tushare nandeng$ python3 crud_test.py Create connection complete 2022-11-04 14:10:38,205 INFO sqlalchemy.engine.Engine SELECT DATABASE() 2022-11-04 14:10:38,206 INFO sqlalchemy.engine.Engine [raw sql] {} Traceback (most recent call last): File "/Users/nandeng/Test/tushare/crud_test.py", line 9, in engine.execute("use " + database_name + ";") File "", line 2, in execute File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 402, in warned return fn(*args, kwargs) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3256, in execute connection = self.connect(close_with_result=True) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect return self._connection_cls(self, close_with_result=close_with_result) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in init else engine.raw_connection() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection return self._wrap_pool_connect(self.pool.connect, _connection) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect return fn() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 320, in connect return _ConnectionFairy._checkout(self) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 486, in checkout rec = pool._do_get() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get with util.safereraise(): File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ raise exception File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get return self._create_connection() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection return _ConnectionRecord(self) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 381, in init self.connect() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 691, in connect )._exec_w_sync_on_first_run(self.dbapi_connection, self) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 320, in _exec_w_sync_on_first_run self(*args, *kw) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in call fn(args, kw) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1695, in go return once_fn(*arg, **kw) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect dialect.initialize(c) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2721, in initialize default.DefaultDialect.initialize(self, connection) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 421, in initialize self.default_isolation_level = self.get_default_isolation_level( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 487, in get_default_isolation_level return self.get_isolation_level(dbapi_conn) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2458, in get_isolation_level if self._is_mysql and self.server_version_info >= (5, 7, 20): TypeError: '>=' not supported between instances of 'NoneType' and 'tuple'

badboynt1 commented 2 years ago

version will be changed to 8.0.30-MatrixOne-v0.6.0 to support this

dengn commented 2 years ago

SQL alchemy CRUD is more or less OK except for creating table with collate and charset.