Altinity / clickhouse-mysql-data-reader

utility to read mysql data
MIT License
369 stars 94 forks source link

Lost connection to MySQL server during query #196

Open fabioaraujopt opened 3 years ago

fabioaraujopt commented 3 years ago

While trying to import data from mysql table to clickhouse, the execution stops with this error:

VALUES
2021-06-18 22:58:56,737/1624057136.737545:DEBUG:Block "" send time: 0.000042
2021-06-18 22:59:40,788/1624057180.788412:DEBUG:Block "" send time: 37.195755
2021-06-18 22:59:40,798/1624057180.798683:DEBUG:Block "" send time: 0.006171
2021-06-18 23:02:19,345/1624057339.345471:CRITICAL:Critical error: (2013, 'Lost connection to MySQL server during query')
2021-06-18 23:02:19,349/1624057339.349685:CRITICAL:Can not migrate table on db=XXX table=XXX
=============
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.7/site-packages/clickhouse_mysql/tablemigrator.py", line 203, in migrate_one_table_data
    rows = self.client.cursor.fetchmany(self.pool_max_rows_num)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 435, in fetchmany
    r = self._fetch_row(size or self.arraysize)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 328, in _fetch_row
    return self._result.fetch_row(size, self._fetch_type)
MySQLdb._exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.7/site-packages/clickhouse_mysql/main.py", line 140, in run
    migrator.migrate_all_tables_data()
  File "/home/ec2-user/.local/lib/python3.7/site-packages/clickhouse_mysql/tablemigrator.py", line 177, in migrate_all_tables_data
    self.migrate_one_table_data(db=db, table=table)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/clickhouse_mysql/tablemigrator.py", line 220, in migrate_one_table_data
    table,
Exception: Can not migrate table on db=ns_dev_data_db table=cdm
=============
Can not migrate table on db=XXX table=XXX
[ec2-user@ip-172-31-19-8 ~]$ 

Any param I can change to import smaller batches or something?