upgrade failed with large ddl jobs and parameter `with-mock-upgrade`

[apollodafoni]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

When there were a large number of DDL jobs and parameter with-mock-upgrade, tiup upgrade failed. TiDB was down during upgrade！

tiup.yaml is as follows:

global:
  arch: amd64
  user: "tidb"
  ssh_port: 22
  deploy_dir: "/tiup/deploy"
  data_dir: "/tiup/data"
  enable_tls: false
server_configs:
  pd: {}
  tidb: {}
  tikv: {}
pd_servers:
  - host: pd-1-peer
  - host: pd-2-peer
  - host: pd-3-peer
tidb_servers:
  - host: tidb-1-peer
  - host: tidb-2-peer
  - host: tidb-3-peer
tikv_servers:
  - host: tikv-1-peer
  - host: tikv-2-peer
  - host: tikv-3-peer
monitoring_servers:
  - host: tiup-peer
    ng_port: 12020
grafana_servers:
  - host: tiup-peer
alertmanager_servers:
  - host: tiup-peer

tiup cluster deploy ddl_upgrade v8.0.0 tiup.yaml --format json -y
tiup cluster start ddl_upgrade --format json -y
sysbench oltp_common --mysql-port=3390 --mysql-user=root --mysql-db=sbtest2 --time=600 --threads=300 --report-interval=10 --db-driver=mysql --tables=100000 --table-size=10 --create_secondary=false prepare
tiup cluster upgrade ddl_upgrade nightly
admin show ddl jobs 1000 where state = 'paused';

2. What did you expect to see? (Required)

upgrade success

3. What did you see instead (Required)

[2024/10/15 16:45:55.363 +08:00] [FATAL] [bootstrap.go:3548] ["mustExecute error"] [error="[schema:1050]Table 'mysql.mock_sys_t_rebase' already exists"] [stack="github.com/pingcap/tidb/pkg/session.mustExecute\n\t/workspace/source/tidb/pkg/session/bootstrap.go:3548\ngithub.com/pingcap/tidb/pkg/session.mockUpgradeToVerLatest\n\t/workspace/source/tidb/pkg/session/mock_bootstrap.go:99\ngithub.com/pingcap/tidb/pkg/session.upgrade\n\t/workspace/source/tidb/pkg/session/bootstrap.go:1554\ngithub.com/pingcap/tidb/pkg/session.runInBootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3686\ngithub.com/pingcap/tidb/pkg/session.bootstrapSessionImpl\n\t/workspace/source/tidb/pkg/session/session.go:3449\ngithub.com/pingcap/tidb/pkg/session.BootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3397\nmain.createStoreAndDomain\n\t/workspace/source/tidb/cmd/tidb-server/main.go:414\nmain.main\n\t/workspace/source/tidb/cmd/tidb-server/main.go:320\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"] [stack="github.com/pingcap/tidb/pkg/session.mustExecute\n\t/workspace/source/tidb/pkg/session/bootstrap.go:3548\ngithub.com/pingcap/tidb/pkg/session.mockUpgradeToVerLatest\n\t/workspace/source/tidb/pkg/session/mock_bootstrap.go:99\ngithub.com/pingcap/tidb/pkg/session.upgrade\n\t/workspace/source/tidb/pkg/session/bootstrap.go:1554\ngithub.com/pingcap/tidb/pkg/session.runInBootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3686\ngithub.com/pingcap/tidb/pkg/session.bootstrapSessionImpl\n\t/workspace/source/tidb/pkg/session/session.go:3449\ngithub.com/pingcap/tidb/pkg/session.BootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3397\nmain.createStoreAndDomain\n\t/workspace/source/tidb/cmd/tidb-server/main.go:414\nmain.main\n\t/workspace/source/tidb/cmd/tidb-server/main.go:320\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"]

4. What is your TiDB version? (Required)

upgrade from v7.1.5/v7.5.3/v8.1.1 to v8.4.0

[apollodafoni]

/assign @tangenta /severity major /label affects-8.4