pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
37.03k stars 5.82k forks source link

Create index is blocked when no /tmp/tidb/tmp_ddl-4000 #45624

Open wengsy150943 opened 1 year ago

wengsy150943 commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

create a index.

2. What did you expect to see? (Required)

successfully create.

3. What did you see instead (Required)

The ddl is blocked.

When kill this ddl, return error mes cannot get disk capacity at /tmp/tidb/tmp_ddl-4000: no such file or directory. And there is no tidb under /tmp.

Index can be created after making dir /tmp/tidb/tmp_ddl-4000 manually .

Since my tidb turns on multi raft, no sure if this bug is related to having multi-raft turned on.

4. What is your TiDB version? (Required)

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                               |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v7.1.0
Edition: Community
Git Commit Hash: 635a4362235e8a3c0043542e629532e3c7bb2756
Git Branch: heads/refs/tags/v7.1.0
UTC Build Time: 2023-05-30 10:58:57
GoVersion: go1.20.3
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
wengsy150943 commented 1 year ago

Another case not sure if it's a bug:

With /tmp/tidb/tmp_ddl-4000 created, create index may also be blocked. Kill ddl and then display err msg:

ERROR 1105 (HY000): DDL job rollback, error msg: sort path: /tmp/tidb/tmp_ddl-4000, disk usage: 227149611008/422621650944, backend usage: 0, please clean up the disk and retry

tiup clean --data is useful in this case, but there exists free disk space before cleaning.

hunao0221 commented 1 year ago

My case is:

[2023-09-05 10:21:44] [HY000][8256] Check ingest environment failed: sort path: /tmp/tidb/tmp_ddl-4000, disk usage: 220939460608/245107195904, backend usage: 0, please clean up the disk and retry