ddl: support concurrent ddl

[hawkingrei]

Signed-off-by: Weizhen Wang wangweizhen@pingcap.com Signed-off-by: xiongjiwei xiongjiwei1996@outlook.com Signed-off-by: wjhuang2016 huangwenjun1997@gmail.com

What problem does this PR solve?

Issue Number: ref https://github.com/pingcap/tidb/issues/32031

this is a big PR and we split it into many commits, every commit almost has a single purpose. I will introduce them briefly, you may need to reference the doc https://github.com/pingcap/tidb/pull/33629

• init ddl tables create tidb_ddl_job, tidb_ddl_reorg, tidb_ddl_history tables with raw meta write, these 3 tables is use to replace the ddl job queue and reorg and history hash table. you can see this part in doc

• setup concurrent ddl env and add ddl worker pool this commit adds the ddl worker pool definition, the ddl job manager will find a job and ship it to a worker in the worker pool. Also, this commit provides a sessionctx wrapper, only use in ddl relate. it just wraps begin, commit and execute.

• add ddl manager to handle ddl job this commit implements the ddl manager, which is used for

  • find a runnable ddl job
  • ship the job to the worker

  you can ref the doc. Also, it adds a function HandleDDLJob which will do the ddl job, and it is a replacer of HandleDDLJobQueue. The last thing of this commit is adding schemaVersionManager to update the schema version in a new txn, it prevents the txn conflict with the schema version key.

• reorg handler for concurrent ddl just implements the partner of the reorg information.

• manage ddl jobs for concurrent ddl same as above, the partner of add job, delete job and many other related to history job

• change ddl interface caller because many of the functions need a session now, we just change the caller

• add metrics for concurrent ddl add metrics

• migrate ddl between table and queue support switch between the old and new ddl framework, migrate the existing ddl job between queue and table

What is changed and how it works?

Check List

Tests

• [x] Unit test
• [ ] Integration test
• [ ] Manual test (add detailed scripts or steps below)
• [ ] No code

Release note

None

[xiongjiwei]

/run-unit-test

[hawkingrei]

/run-all-tests

[hawkingrei]

/run-unit-test

[hawkingrei]

/run-unit-test

[hawkingrei]

/run-check_dev

[xiongjiwei]

/run-unit-test

[hawkingrei]

/run-check_dev_2

[xiongjiwei]

/run-unit-test

[hawkingrei]

/run-unit-test

[hawkingrei]

/run-all-tests

[hawkingrei]

/run-all-tests

[hawkingrei]

/run-all-tests

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/7b9cd33e861095a82a27b272e6c9fb57ae5ae647

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e2908a7c37db2daddea6802fdf419333b8c75159

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/699dd14a931f8677cd88843d6c7d3ade16818cbf

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/04305908b7545c69530de84027f527bc20375606

[hawkingrei]

/run-mysql-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/9dab8fd2822b16ee73261727ccbffceccb70a04e

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/37adf2f98e4a46063017bb37a75d95df5e3ec8c8

[hawkingrei]

/run-build

[hawkingrei]

/run-all-tests

[xiongjiwei]

/run-unit-test

[hawkingrei]

/run-build

[hawkingrei]

/run-mysql-test

[hawkingrei]

/run-unit-test

[xiongjiwei]

the timeout happens because bootstrap takes more than 10s in the test.

[hawkingrei]

/run-unit-test

[hawkingrei]

/run-unit-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/bbc20e62d82f6e03ec5eb4fa51e0ec26986a3531

[xiongjiwei]

all tests passed!

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/ea199dca1ca23d923f5563b1b61f470535ffa042

[xiongjiwei]

/run-check_dev_2

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/56a5a2847471020f000811c41c7f75521ac2ce83

[hawkingrei]

/run-mysql-test

[hawkingrei]

/run-unit-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e0455b7877f8f60225691503e4779b29914c79e9

[hawkingrei]

all tests passed!

all tests passed again!

[hawkingrei]

/run-unit-test

[hawkingrei]

/run-unit-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/19dfe59a27737f2b1669d235c2c824e07ac05eeb

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/416ab6cfc2e730708720f0d09158db602ce9d901

[hawkingrei]

force push for syncing merge.

[xiongjiwei]

/run-unit-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/ef6f364ad4f44cd90d5165da081de5f67086d015

[xiongjiwei]

/run-unit-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/7ed2685368953e6b258ff453f18d57a3e27618aa

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/4b5341ce3d50b47e9259172042fb87d94e6e1cfe

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/1927197ed5d11969149eaf051c0fb9ec238c66d4

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/b87229a27dbc22a575448ee20d86a20e9e2a160c

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/c3974257cfc7ef41a024446441522516d08ac97e