ddl: support concurrent ddl - Githubissues

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/

https://pingcap.com

Apache License 2.0

36.49k stars 5.74k forks source link

ddl: support concurrent ddl #32169

Closed hawkingrei closed 1 year ago

hawkingrei commented 2 years ago

Signed-off-by: Weizhen Wang wangweizhen@pingcap.com Signed-off-by: xiongjiwei xiongjiwei1996@outlook.com Signed-off-by: wjhuang2016 huangwenjun1997@gmail.com

What problem does this PR solve?

Issue Number: ref https://github.com/pingcap/tidb/issues/32031

this is a big PR and we split it into many commits, every commit almost has a single purpose. I will introduce them briefly, you may need to reference the doc https://github.com/pingcap/tidb/pull/33629

init ddl tables create tidb_ddl_job, tidb_ddl_reorg, tidb_ddl_history tables with raw meta write, these 3 tables is use to replace the ddl job queue and reorg and history hash table. you can see this part in doc
setup concurrent ddl env and add ddl worker pool this commit adds the ddl worker pool definition, the ddl job manager will find a job and ship it to a worker in the worker pool. Also, this commit provides a sessionctx wrapper, only use in ddl relate. it just wraps begin, commit and execute.
add ddl manager to handle ddl job this commit implements the ddl manager, which is used for
- find a runnable ddl job
- ship the job to the worker
you can ref the doc. Also, it adds a function HandleDDLJob which will do the ddl job, and it is a replacer of HandleDDLJobQueue. The last thing of this commit is adding schemaVersionManager to update the schema version in a new txn, it prevents the txn conflict with the schema version key.
reorg handler for concurrent ddl just implements the partner of the reorg information.
manage ddl jobs for concurrent ddl same as above, the partner of add job, delete job and many other related to history job
change ddl interface caller because many of the functions need a session now, we just change the caller
add metrics for concurrent ddl add metrics
migrate ddl between table and queue support switch between the old and new ddl framework, migrate the existing ddl job between queue and table

What is changed and how it works?

Check List

Tests

[x] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Release note

None

hawkingrei commented 2 years ago

/run-build

hawkingrei commented 2 years ago

/run-all-tests

hawkingrei commented 2 years ago

/run-unit-test

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/7eaba28950553f07033c3dc9fca12c35095e00da

hawkingrei commented 2 years ago

/run-unit-test

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/62c38c703376600a3828f54f1621428fa72dfaf9

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/073acd500e80b2edb6f6714ef5bdc28be5db0f7c

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/eeda355c476445e994b5ceea87fd71d4930b8f52

xiongjiwei commented 2 years ago

@zimulala some comments are addressed at the newest commit, PTAL. And I will add some UT in another commit

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e98200dd9944b738eed0c2d0763a2726f889a927

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/70b44d747d224310f03ead83f4b8a4cc11ec73e5

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/c09a8399de45c15cf7b027281492a5a8363fe612

sre-bot commented 2 years ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/83c3cdae2ef3e83307f0cda8e46fd98a787d7033

xiongjiwei commented 2 years ago

@zimulala @tangenta PTAL

xiongjiwei commented 2 years ago

/run-integration-ddl-test

xiongjiwei commented 1 year ago

/re-build

xiongjiwei commented 1 year ago

/run-integration-ddl-test

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/b1184d9b1f04090e6a587c382378302a15679c57

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/f057fc91b1ac514d6bba13c9a3951889de6bd2fe

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/02e7d0404591691dd52a9563565468dc1a03bb62

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a48e61587ace78af5f2721080cccdc8aa611b74a

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/701f7f2b12ac2fede500fe1f71dc490eb09b8e84

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a083f48b7c98af62c46dd4f960c75e95d0a8b202

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/cc35134b40998c8439667a85b81e31b78ef39223

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/d70aaae994380b83fcfcb186ec7cf98465acaf61

xiongjiwei commented 1 year ago

/run-mysql-test

xiongjiwei commented 1 year ago

/run-integration-common-test

xiongjiwei commented 1 year ago

/run-integration-ddl-test

xiongjiwei commented 1 year ago

/run-tics-test

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/4e14cb832c26d37eed4b898c33ff5af33bd35205

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/093130faacbddf0fda3198a7471c6d891370d62e

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/213b0bb0e68a9baef9da92f25c16124175477f4f

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/462d3d43de0744e45a07a9ea0dafa26d382417db

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/4d65f2f927e7a315d99b65da9c6a7ff29ac9c8cd

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/96f934128bc09da17ada9d66421789d0b5786dc2

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/42dce41ca682dcdb81af7b94aef343898bdda6df

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/132a4e6a4ec9c5e9d655d36c83fa168228637cb8

wjhuang2016 commented 1 year ago

/run-all-tests

xiongjiwei commented 1 year ago

/run-all-tests

xiongjiwei commented 1 year ago

/run-check_dev

xiongjiwei commented 1 year ago

/run-check_dev

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/fd21f7cb07e445da0b53c2d2ac105e8c7423c135

hawkingrei commented 1 year ago

/run-integration-ddl-test

xiongjiwei commented 1 year ago

/run-integration-ddl-test

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a4053d41883d19c17fe889e3c12d963023595733

xiongjiwei commented 1 year ago

/run-all-tests

sre-bot commented 1 year ago

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/a4053d41883d19c17fe889e3c12d963023595733

hawkingrei commented 1 year ago

/run-unit-test

hawkingrei commented 1 year ago

/run-check_dev

hawkingrei commented 1 year ago

/run-unit-test

Previous Next