pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.2k stars 5.84k forks source link

enable concurrent ddl, let tidb.time > pd.time and run, create database failed. #36642

Closed vivid392845427 closed 2 years ago

vivid392845427 commented 2 years ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

run utf-go-partition case and check the result,the use case steps are as follows: 1.enable concurrent ddl 2.enable failpoint // make the timestamp from pd greater than local time in tidb with mock opertion Try(cli.EnableFailpoint(resource.PD, "github.com/tikv/pd/server/tso/fallBackUpdate", "return")) 3.create database if not exists dbname ---check

run failed on tcms:https://tcms.pingcap.net/dashboard/executions/plan/1021930

2. What did you expect to see? (Required)

database create success.

3. What did you see instead (Required)

database create failed. log as follow.if cuncurrent ddl set disable, case run success. {"pod":"tidb-partition-tidb-0","log":"[conn.go:1165] [\"command dispatched failed\"] [conn=1626089736550483561] [connInfo=\"id:1626089736550483561, addr:10.233.99.135:33772 status:10, collation:utf8mb4_general_ci, user:root\"] [command=Query] [status=\"inTxn:0, autocommit:1\"] [sql=\"create database if not exists TICASE_1213_RANGE_NvdHPoSG\"] [txn_mode=PESSIMISTIC] [timestamp=434814812425814025] [err=\"[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(twoPhaseCommitter).checkSchemaValid\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.1-0.20220721031657-e38d2b07de3f/txnkv/transaction/2pc.go:1830\ngithub.com/tikv/client-go/v2/txnkv/transaction.(twoPhaseCommitter).execute\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.1-0.20220721031657-e38d2b07de3f/txnkv/transaction/2pc.go:1510\ngithub.com/tikv/client-go/v2/txnkv/transaction.(KVTxn).Commit\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.1-0.20220721031657-e38d2b07de3f/txnkv/transaction/txn.go:448\ngithub.com/pingcap/tidb/store/driver/txn.(tikvTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:80\ngithub.com/pingcap/tidb/session.(LazyTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txn.go:393\ngithub.com/pingcap/tidb/session.(session).commitTxnWithTemporaryData\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:761\ngithub.com/pingcap/tidb/session.(session).doCommit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:651\ngithub.com/pingcap/tidb/session.(session).doCommitWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:893\ngithub.com/pingcap/tidb/session.(session).CommitTxn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:959\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:280\ngithub.com/pingcap/tidb/session.finishStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:246\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2144\ngithub.com/pingcap/tidb/session.(session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1999\ngithub.com/pingcap/tidb/session.(session).ExecuteInternal\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1503\ngithub.com/pingcap/tidb/ddl.(session).execute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl.go:1616\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/job_table.go:276\ngithub.com/pingcap/tidb/ddl.(ddl).addBatchDDLJobs2Table\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:405\ngithub.com/pingcap/tidb/ddl.(ddl).addBatchDDLJobs\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:298\ngithub.com/pingcap/tidb/ddl.(ddl).limitDDLJobs\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:286\ngithub.com/pingcap/tidb/util.(WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/error/error.go:169\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/error.go:154\ngithub.com/pingcap/tidb/store/driver/txn.(tikvTxn).extractKeyErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:294\ngithub.com/pingcap/tidb/store/driver/txn.(tikvTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:81\ngithub.com/pingcap/tidb/session.(LazyTxn).Commit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/txn.go:393\ngithub.com/pingcap/tidb/session.(session).commitTxnWithTemporaryData\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:761\ngithub.com/pingcap/tidb/session.(session).doCommit\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:651\ngithub.com/pingcap/tidb/session.(session).doCommitWithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:893\ngithub.com/pingcap/tidb/session.(session).CommitTxn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:959\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:280\ngithub.com/pingcap/tidb/session.finishStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/tidb.go:246\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2144\ngithub.com/pingcap/tidb/session.(session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1999\ngithub.com/pingcap/tidb/session.(session).ExecuteInternal\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1503\ngithub.com/pingcap/tidb/ddl.(session).execute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl.go:1616\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/job_table.go:276\ngithub.com/pingcap/tidb/ddl.(ddl).addBatchDDLJobs2Table\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:405\ngithub.com/pingcap/tidb/ddl.(ddl).addBatchDDLJobs\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:298\ngithub.com/pingcap/tidb/ddl.(ddl).limitDDLJobs\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/ddl/ddl_worker.go:286\ngithub.com/pingcap/tidb/util.(WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\"]","container":"tidb","level":"INFO","namespace":"oltp-tps-992606-1-620"}

4. What is your TiDB version? (Required)

version="Release Version: v6.2.0-dirty\nEdition: Community\nGit Commit Hash: 121a6669a85cdb8da942de466d42d5dfdecc0b7d\nGit Branch: heads/refs/tags/v6.2.0\nUTC Build Time: 2022-07-22 03:12:58\nGoVersion: go1.18.2\nRace Enabled: false\nTiKV Min Version: 6.2.0-alpha\nCheck Table Before Drop: false\nStore: tikv"

wjhuang2016 commented 2 years ago

It's not a bug, the test should be refined after supporting concurrent DDL.