This PR introduces multi-threaded replication for applying DML queries to the ghost table. The goal is to be able to migrate tables with high rate of DML queries (e.g. >5k rows/s). Currently gh-ost lags behind in these situations, taking a very long time to complete or not completing at all.
Similar to MySQL replication threads, gh-ost will stream binlog events from the source and group them into transactions. It then submits the transactions to a pool of workers to apply the transactions concurrently on the ghost table. We ensure that dependent transactions are applied in a consistent order (equivalent to MySQL multi-threaded replication with replica_preserve_commit_order=0).
With WRITESET enabled on the source, this enables a great amount of parallelism in the transaction applier.
Changes
TODO
Performance tests
TODO
In case this PR introduced Go code changes:
[x] contributed code is using same conventions as original code
[x] script/cibuild returns with no formatting errors, build errors or unit test errors.
Description
This PR introduces multi-threaded replication for applying DML queries to the ghost table. The goal is to be able to migrate tables with high rate of DML queries (e.g. >5k rows/s). Currently gh-ost lags behind in these situations, taking a very long time to complete or not completing at all.
Similar to MySQL replication threads, gh-ost will stream binlog events from the source and group them into transactions. It then submits the transactions to a pool of workers to apply the transactions concurrently on the ghost table. We ensure that dependent transactions are applied in a consistent order (equivalent to MySQL multi-threaded replication with
replica_preserve_commit_order=0). WithWRITESETenabled on the source, this enables a great amount of parallelism in the transaction applier.Changes
TODO
Performance tests
TODO
script/cibuildreturns with no formatting errors, build errors or unit test errors.