search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.25k stars 5.84k forks source link

ddl rollingback is freezed #45746

Open chinesezhaoyun opened 1 year ago

chinesezhaoyun commented 1 year ago

Bug Report

1. Minimal reproduce step (Required)

Cancel ddl state is freezed.

Asktug DDL rollingback 阻塞 Asktug 批量执行DDL处于done状态

Step as follow. But I can't reproduce the issue.

Enable variables SET GLOBAL tidb_enable_metadata_lock = 'ON' ; SET GLOBAL tidb_ddl_enable_fast_reorg = 'ON' ;

  1. add index on big table
  2. cancel add index
  3. scale-out a tidb node
  4. scale-in a tidb node
  5. restart tidb node
+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+
| JOB_ID | DB_NAME                  | TABLE_NAME                          | JOB_TYPE                  | SCHEMA_STATE          | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE       |
+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+
|  23333 | spider_company_entity    | spider_enterprise_regno_sd_info     | add index /* txn-merge */ | delete reorganization |     16417 |    16627 | 142555210 | 2023-07-21 14:43:55 | 2023-07-21 14:43:55 | NULL                | rollingback |
|  23335 | spider_company_entity    | spider_enterprise_regno_sd_gt_info  | add index /* txn-merge */ | delete reorganization |     16417 |    16651 |  10023574 | 2023-07-21 14:59:12 | 2023-07-21 14:59:12 | NULL                | rollingback |
|  23357 | spider_company_entity    | spider_enterprise_regno_sd_info     | add index                 | none                  |     16417 |    16627 |         0 | 2023-07-21 16:26:56 | NULL                | NULL                | cancelling  |
|  24256 | spider_company_entity    | spider_enterprise_regno_sd_gt_info  | rename table              | none                  |     16417 |    16651 |         0 | 2023-08-01 18:14:06 | NULL                | NULL                | cancelling  |
|  24316 | spider_company_test      | t_proposed_notice_structural_middle | modify column             | public                |      7200 |    23199 |         0 | 2023-08-02 11:09:29 | 2023-08-02 11:09:29 | 2023-08-02 11:09:29 | synced      |
|  24315 | spider_company_base_info | t_proposed_notice_structural_middle | modify column             | public                |     13300 |    24191 |         0 | 2023-08-02 11:09:14 | 2023-08-02 11:09:14 | 2023-08-02 11:09:15 | synced      |
+--------+--------------------------+-------------------------------------+---------------------------+-----------------------+-----------+----------+-----------+---------------------+---------------------+---------------------+-------------+

tidb log

[2023/08/02 13:53:01.117 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.139 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.161 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.183 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23335] [ver=17987]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id 82ad37d5-2f61-4616-af37-47cc32984a59"] ["ddl id"=23333] [ver=17988]
[2023/08/02 13:53:01.213 +08:00] [INFO] [syncer.go:333] ["[ddl] syncer check all versions, someone is not synced"] [info="instance ip xx.xx.xx.xx, port 5000, id e8f6e8b5-c2fa-48c1-94e5-89094a9b7729"] ["ddl id"=23333] [ver=17988]

2. What did you expect to see? (Required)

ddl state is synced.

3. What did you see instead (Required)

Create new table and load data.

4. What is your TiDB version? (Required)

Release Version: v6.5.3 Edition: Community Git Commit Hash: 71e6696f020ccf42408976d4a82cafb95717a2e4 Git Branch: heads/refs/tags/v6.5.3 UTC Build Time: 2023-06-12 13:52:32 GoVersion: go1.19.9 Race Enabled: false TiKV Min Version: 6.2.0-alpha Check Table Before Drop: false Store: tikv

wjhuang2016 commented 1 year ago

@chinesezhaoyun Hi, Can it recover after some minutes?

chinesezhaoyun commented 1 year ago

@wjhuang2016 It still hasn't recovered. How can I fix it.

wjhuang2016 commented 1 year ago

@wjhuang2016 It still hasn't recovered. How can I fix it.

@chinesezhaoyun I replied to you in Asktug 批量执行DDL处于done状态, we can share the information there.