search

Altinity / clickhouse-mysql-data-reader

utility to read mysql data
MIT License
368 stars 94 forks source link

Data Discrepancy Between MySQL And ClickHouse? #189

Closed sv3t0sl4v closed 3 years ago

sv3t0sl4v commented 3 years ago

Greetings!

Using clickhouse-mysql-data-reader. The counts between the two are different. What could possibly be best settings for migrating and pumping new data from MySQL to ClickHouse, if the source MySQL table is constantly being written to and read from? Difference is huge. like tens of thousands of rows from MySQL constantly missing from ClickHouse. Have to make it pump data faster. Any ideas how?

Regards!

sv3t0sl4v commented 3 years ago

Insane... Added secondary index, ran optimize table x final and voilà. No more differences in data on MySQL and ClickHouse and queries are 80+ times faster.

sv3t0sl4v commented 3 years ago

Strange. Had to migrate and keep pumping data anew because of changed table structure. Looks like records are there, but counts do not mach.

sv3t0sl4v commented 3 years ago

Update... Only on the import date data has discrepancy. Difference is about 28k rows. Everything else seems normal. Optimizing table did not fix the count this time.

sv3t0sl4v commented 3 years ago

I think it is the rows being generated while migration. Will replace the slice for day with discrepancy.