[Bug]: [date 9.5]tke regression:cn restart caused panic error invalid free, mp header corruption

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch Name

main

Commit ID

a81e4b77e

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/10722505642/job/29762092051

企业微信截图_11cfdabc-015b-4ecf-8cbc-b7b4a6e0fc7c

cn restart这个时间段，在执行如下操作 alter table sysbench_db.sbtest$i rename to sysbench_db.bak_sbtest$i;create table sysbench_db.sbtest$i(id int not null,k int default 0,c char(120) default '',pad char(60) default '')cluster by(id);insert into sysbench_db.sbtest$i select from sysbench_db.bak_sbtest$i;drop table sysbench_db.bak_sbtest$i" 客户端确认了表的ddl，应该是在insert into sysbench_db.sbtest$i select from sysbench_db.bak_sbtest$ load数据时挂掉了

panic error： {"level":"ERROR","time":"2024/09/06 04:55:10.959768 +0000","caller":"mpool/mpool.go:630","msg":"error: internal error: invalid free, mp header corruption"} {"level":"ERROR","time":"2024/09/06 04:55:10.959967 +0000","caller":"mpool/mpool.go:630","msg":"error: internal error: invalid free, mp header corruption"} 2024/09/06 04:55:10.960184 [ants]: worker exits from panic: internal error: invalid free, mp header corruption goroutine 3278917 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:26 +0x5e github.com/panjf2000/ants/v2.(goWorker).run.func1.1() /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:56 +0x85 panic({0x4889040?, 0xc581d599c0?}) /usr/local/go/src/runtime/panic.go:785 +0x132 github.com/matrixorigin/matrixone/pkg/common/mpool.(MPool).Free(0xc05291fa40?, {0xc053624018?, 0x0?, 0xc36c3cb020?}) /go/src/github.com/matrixorigin/matrixone/pkg/common/mpool/mpool.go:630 +0x339 github.com/matrixorigin/matrixone/pkg/container/vector.(Vector).Free(0xc2402a5000, 0xc05291fa40) /go/src/github.com/matrixorigin/matrixone/pkg/container/vector/vector.go:477 +0x39 github.com/matrixorigin/matrixone/pkg/container/batch.(Batch).Clean(0xc05507f570, 0xc05291fa40) /go/src/github.com/matrixorigin/matrixone/pkg/container/batch/batch.go:281 +0x79 github.com/matrixorigin/matrixone/pkg/sql/colexec/table_scan.(TableScan).Free(0xc22dcf8900, 0x2?, 0x4f?, {0xc05b7d3b28?, 0x476105?}) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/table_scan/types.go:101 +0x45 github.com/matrixorigin/matrixone/pkg/vm/pipeline.(Pipeline).Cleanup.func2({0x2?, 0x0?}, {0x5520c40?, 0xc22dcf8900?}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/types.go:104 +0x38 github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x55205c0, 0xc1648e7b90}, {0x5520c40, 0xc22dcf8900}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:333 +0xe9 github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x55205c0, 0xc0b91ce000}, {0x55205c0, 0xc1648e7b90}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:329 +0xbe github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x551fcd0, 0xc23d2bf360}, {0x55205c0, 0xc0b91ce000}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:329 +0xbe github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x551fc68, 0xc07ffb6b40}, {0x551fcd0, 0xc23d2bf360}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:329 +0xbe github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x551fb30, 0xc40ebd4360}, {0x551fc68, 0xc07ffb6b40}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:329 +0xbe github.com/matrixorigin/matrixone/pkg/vm.doHandleAllOp({0x0, 0x0}, {0x551fb30, 0xc40ebd4360}, 0xc05b7d3c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:329 +0xbe github.com/matrixorigin/matrixone/pkg/vm.HandleAllOp(...) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:337 github.com/matrixorigin/matrixone/pkg/vm/pipeline.(Pipeline).Cleanup(0xc05b7d3e40, 0xc405ea8c00, 0x1, 0x0, {0x547ee20, 0xc581d59880}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/types.go:103 +0x136 github.com/matrixorigin/matrixone/pkg/sql/compile.(Scope).Run.func1() /go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/scope.go:167 +0x2dd github.com/matrixorigin/matrixone/pkg/sql/compile.(Scope).Run(0xc0b32e8460, 0x10000c01e99bfa0?) /go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/scope.go:210 +0x313 github.com/matrixorigin/matrixone/pkg/sql/compile.(Scope).MergeRun.func1() /go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/scope.go:267 +0x7f github.com/panjf2000/ants/v2.(goWorker).run.func1() /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:67 +0x8a created by github.com/panjf2000/ants/v2.(goWorker).run in goroutine 3278846 /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:48 +0x5c

runtime: nameOff 0x6807a76b base 0x40590b not in ranges: types 0x3ff6020 etypes 0x573313d fatal error: runtime: name offset base pointer out of range

goroutine 3278919 gp=0xc0c71c6700 m=5 mp=0xc000100008 [running]: runtime.throw({0x4cbc360?, 0xc0000fccf0?}) /usr/local/go/src/runtime/panic.go:1067 +0x48 fp=0xc0531e8ff0 sp=0xc0531e8fc0 pc=0x47b528 runtime.resolveNameOff(0x0?, 0x6807a76b) /usr/local/go/src/runtime/type.go:143 +0x1bb fp=0xc0531e9048 sp=0xc0531e8ff0 pc=0x4810bb runtime.rtype.nameOff(...) /usr/local/go/src/runtime/type.go:149 runtime.rtype.string({0x422c0e0?}) /usr/local/go/src/runtime/type.go:26 +0x1b fp=0xc0531e9070 sp=0xc0531e9048 pc=0x46fe9b runtime.(TypeAssertionError).Error(0xc4302fe2d0) /usr/local/go/src/runtime/error.go:42 +0x90 fp=0xc0531e9178 sp=0xc0531e9070 pc=0x411a30 fmt.(pp).handleMethods(0xc1e537e270, 0x1?) /usr/local/go/src/fmt/print.go:667 +0x3e3 fp=0xc0531e9338 sp=0xc0531e9178 pc=0x5253e3 fmt.(pp).printArg(0xc1e537e270, {0x4430f20, 0xc4302fe2d0}, 0x76) /usr/local/go/src/fmt/print.go:756 +0x446 fp=0xc0531e93d0 sp=0xc0531e9338 pc=0x525c26 fmt.(pp).doPrintf(0xc1e537e270, {0x4c2b4eb, 0xd}, {0xc0531e9550, 0x2, 0x2}) /usr/local/go/src/fmt/print.go:1074 +0x37e fp=0xc0531e94b8 sp=0xc0531e93d0 pc=0x5286de fmt.Sprintf({0x4c2b4eb, 0xd}, {0xc0531e9550, 0x2, 0x2}) /usr/local/go/src/fmt/print.go:239 +0x53 fp=0xc0531e9510 sp=0xc0531e94b8 pc=0x522833 github.com/matrixorigin/matrixone/pkg/common/moerr.ConvertPanicError({0x54c33b0, 0xc525ff41e0}, {0x4430f20, 0xc4302fe2d0}) /go/src/github.com/matrixorigin/matrixone/pkg/common/moerr/error.go:636 +0xae fp=0xc0531e9580 sp=0xc0531e9510 pc=0xa238ce github.com/matrixorigin/matrixone/pkg/vm/pipeline.catchPanic({0x54c33b0, 0xc525ff41e0}, 0xc0531ebd90) /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/pipeline.go:103 +0x4c fp=0xc0531e95c8 sp=0xc0531e9580 pc=0x3769d2c github.com/matrixorigin/matrixone/pkg/vm/pipeline.(Pipeline).run.deferwrap1() /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/pipeline.go:86 +0x28 fp=0xc0531e95f0 sp=0xc0531e95c8 pc=0x3769ca8 panic({0x4430f20?, 0xc4302fe2d0?}) /usr/local/go/src/runtime/panic.go:785 +0x132 fp=0xc0531e96a0 sp=0xc0531e95f0 pc=0x47b232 runtime.panicdottypeE(0x40590b, 0x4516980, 0x4322600) /usr/local/go/src/runtime/iface.go:275 +0x65 fp=0xc0531e96c0 sp=0xc0531e96a0 pc=0x4146e5 github.com/matrixorigin/matrixone/pkg/common/malloc.(fixedSizeMmapAllocator).Allocate(0xc0024ab8c0, 0x0) /go/src/github.com/matrixorigin/matrixone/pkg/common/malloc/fixed_size_mmap_allocator.go:162 +0x15b fp=0xc0531e9770 sp=0xc0531e96f8 pc=0xbe597b github.com/matrixorigin/matrixone/pkg/common/malloc.(ClassAllocator[...]).Allocate(0x491a85?, 0xefde?, 0xc0531e97f0) /go/src/github.com/matrixorigin/matrixone/pkg/common/malloc/class_allocator.go:54 +0x29 fp=0xc0531e97e8 sp=0xc0531e97b8 pc=0xbeeec9 github.com/matrixorigin/matrixone/pkg/common/malloc.ShardedAllocator.Allocate({0xc00073dd00, 0xe, 0xc000100508?}, 0xefde, 0x0) /go/src/github.com/matrixorigin/matrixone/pkg/common/malloc/sharded_allocator.go:32 +0x6c fp=0xc0531e9870 sp=0xc0531e9840 pc=0xbe780c github.com/matrixorigin/matrixone/pkg/common/malloc.(ShardedAllocator).Allocate(0xc0531e98f0?, 0x44e1c6?, 0x0?) /go/src/github.com/matrixorigin/matrixone/pkg/common/malloc/random_allocator.go:40 +0x52 fp=0xc0531e98d0 sp=0xc0531e98a8 pc=0xbe7192 github.com/matrixorigin/matrixone/pkg/common/malloc.(MetricsAllocator).Allocate(0xc000b7ac60, 0xefde, 0x40d286?) /go/src/github.com/matrixorigin/matrixone/pkg/common/malloc/metrics_allocator.go:74 +0x2c fp=0xc0531e9988 sp=0xc0531e9930 pc=0xbe696c github.com/matrixorigin/matrixone/pkg/common/malloc.(RandomAllocator).Allocate(0xc003bff6b0, 0xefde, 0x0) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/bytes.go:53 +0x1d fp=0xc0531e9a00 sp=0xc0531e99b0 pc=0x12e313d github.com/matrixorigin/matrixone/pkg/objectio.constructorFactory.func1({0x547f7a0?, 0xc4302fe2a0?}, {0x7ff7172f7000?, 0x0?, 0x0?}, {0x547fa20?, 0xc002390610?}) /go/src/github.com/matrixorigin/matrixone/pkg/objectio/constructors.go:46 +0x112 fp=0xc0531e9a90 sp=0xc0531e9a00 pc=0x219b272 github.com/matrixorigin/matrixone/pkg/fileservice.(IOEntry).setCachedData(0xc05ae04f08, {0x54c3378, 0xc4302fe1b0}) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/io_entry.go:46 +0x205 fp=0xc0531e9b60 sp=0xc0531e9a90 pc=0x12ee405 github.com/matrixorigin/matrixone/pkg/fileservice.(IOEntry).ReadFromOSFile(0xc05ae04f08, {0x54c3378, 0xc4302fe1b0}, 0xc4e811e800) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/io_entry.go:78 +0x339 fp=0xc0531e9c38 sp=0xc0531e9b60 pc=0x12ee9b9 github.com/matrixorigin/matrixone/pkg/fileservice.(DiskCache).Read.func3(0xc05ae04f08) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/disk_cache.go:308 +0x613 fp=0xc0531e9e60 sp=0xc0531e9c38 pc=0x12e6c33 github.com/matrixorigin/matrixone/pkg/fileservice.(DiskCache).Read(0xc002398780, {0x54c3378, 0xc4302fe1b0}, 0xc2a075ae40) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/disk_cache.go:320 +0x416 fp=0xc0531ea1f0 sp=0xc0531e9e60 pc=0x12e61f6 github.com/matrixorigin/matrixone/pkg/fileservice.readCache({0x54c3378?, 0xc4302fe1b0?}, {0x54d7140, 0xc002398780?}, 0x0?) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/cache.go:181 +0x13e fp=0xc0531ea420 sp=0xc0531ea1f0 pc=0x12e3b1e github.com/matrixorigin/matrixone/pkg/fileservice.(S3FS).Read(0xc0026fc580, {0x54c33b0, 0xc20c5bb770}, 0xc2a075ae40) /go/src/github.com/matrixorigin/matrixone/pkg/fileservice/s3fs.go:538 +0xc37 fp=0xc0531ea650 sp=0xc0531ea420 pc=0x130d897 /go/src/github.com/matrixorigin/matrixone/pkg/objectio/funcs.go:202 +0x794 fp=0xc0531ea900 sp=0xc0531ea650 pc=0x219cb74 github.com/matrixorigin/matrixone/pkg/objectio.ReadOneBlock(...) /go/src/github.com/matrixorigin/matrixone/pkg/objectio/funcs.go:135 /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/funcs.go:48 +0x1d8 fp=0xc0531ea9f0 sp=0xc0531ea900 pc=0x27624b8 /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/funcs.go:156 +0xa5 fp=0xc0531eaa80 sp=0xc0531ea9f0 pc=0x27632c5 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.readBlockData.func1({0xc402550070?, 0x18?, 0xc0531eabb8?}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/read.go:518 +0x165 fp=0xc0531eab50 sp=0xc0531eaa80 pc=0x276ac85 /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/read.go:560 +0x249 fp=0xc0531eac90 sp=0xc0531eab50 pc=0x276a629 /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/read.go:351 +0x125 fp=0xc0531eaf18 sp=0xc0531eac90 pc=0x2769385 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio.BlockDataRead({0x54c33b0, 0xc20c5bb770}, {, }, , {, }, {0xc402550070, 0x4, 0x4}, ...) /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/blockio/read.go:219 +0x69b fp=0xc0531eb0c0 sp=0xc0531eaf18 pc=0x276841b /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/reader.go:375 +0x557 fp=0xc0531eb308 sp=0xc0531eb0c0 pc=0x335f657 github.com/matrixorigin/matrixone/pkg/sql/colexec/table_scan.(TableScan).Call(0xc19d6ee500, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/table_scan/table_scan.go:112 +0x43d fp=0xc0531eb6a0 sp=0xc0531eb308 pc=0x368947d github.com/matrixorigin/matrixone/pkg/vm.ChildrenCall({0x5520c40, 0xc19d6ee500}, 0xc405ea8c80, {0x554a3a0, 0xc3757b0280}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:258 +0x5a fp=0xc0531eb708 sp=0xc0531eb6a0 pc=0x2f4393a github.com/matrixorigin/matrixone/pkg/sql/colexec/projection.(Projection).Call(0xc1648e7c20, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/projection/projection.go:62 +0x1a5 fp=0xc0531eb7e8 sp=0xc0531eb708 pc=0x37067a5 github.com/matrixorigin/matrixone/pkg/vm.ChildrenCall({0x55205c0, 0xc1648e7c20}, 0xc405ea8c80, {0x554a3a0, 0xc3757b0240}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:258 +0x5a fp=0xc0531eb850 sp=0xc0531eb7e8 pc=0x2f4393a github.com/matrixorigin/matrixone/pkg/sql/colexec/projection.(Projection).Call(0xc1648e7d40, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/projection/projection.go:62 +0x1a5 fp=0xc0531eb930 sp=0xc0531eb850 pc=0x37067a5 github.com/matrixorigin/matrixone/pkg/vm.ChildrenCall({0x55205c0, 0xc1648e7d40}, 0xc405ea8c80, {0x554a3a0, 0xc3757b0180}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:258 +0x5a fp=0xc0531eb998 sp=0xc0531eb930 pc=0x2f4393a github.com/matrixorigin/matrixone/pkg/sql/colexec/preinsert.(PreInsert).Call(0xc23d2bf220, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/preinsert/preinsert.go:110 +0x1a9 fp=0xc0531ebaa0 sp=0xc0531eb998 pc=0x36f8529 github.com/matrixorigin/matrixone/pkg/vm.ChildrenCall({0x551fcd0, 0xc23d2bf220}, 0xc405ea8c80, {0x554a3a0, 0xc3ed1b93c0}) /go/src/github.com/matrixorigin/matrixone/pkg/vm/types.go:258 +0x5a fp=0xc0531ebb08 sp=0xc0531ebaa0 pc=0x2f4393a github.com/matrixorigin/matrixone/pkg/sql/colexec/insert.(Insert).insert_s3(0xc07ffb69c0, 0xc405ea8c80, {0x554a3a0, 0xc3ed1b93c0}) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/insert/insert.go:119 +0x394 fp=0xc0531ebc08 sp=0xc0531ebb08 pc=0x363bbf4 github.com/matrixorigin/matrixone/pkg/sql/colexec/insert.(Insert).Call(0xc07ffb69c0, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/insert/insert.go:106 +0x192 fp=0xc0531ebcb0 sp=0xc0531ebc08 pc=0x363b6d2 github.com/matrixorigin/matrixone/pkg/sql/colexec/connector.(Connector).Call(0xc40ebd4840, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/sql/colexec/connector/connector.go:47 +0x6e fp=0xc0531ebd70 sp=0xc0531ebcb0 pc=0x34a6cce github.com/matrixorigin/matrixone/pkg/vm/pipeline.(Pipeline).run(0xc0531ebe40, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/pipeline.go:91 +0xdf fp=0xc0531ebdd8 sp=0xc0531ebd70 pc=0x3769bbf github.com/matrixorigin/matrixone/pkg/vm/pipeline.(Pipeline).Run(0xc01e034e40, {0x54d7800, 0xc05413b380}, 0x0, 0xc405ea8c80) /go/src/github.com/matrixorigin/matrixone/pkg/vm/pipeline/pipeline.go:61 +0xd1 fp=0xc0531ebdf8 sp=0xc0531ebdd8 pc=0x3769951 github.com/matrixorigin/matrixone/pkg/sql/compile.(Scope).Run(0xc0b32e81c0, 0xc00294e7a0?) /go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/scope.go:202 +0x225 fp=0xc0531ebee0 sp=0xc0531ebdf8 pc=0x37eda65 github.com/matrixorigin/matrixone/pkg/sql/compile.(Scope).MergeRun.func1() /go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/scope.go:267 +0x7f fp=0xc0531ebf88 sp=0xc0531ebee0 pc=0x37ee95f github.com/panjf2000/ants/v2.(goWorker).run.func1() /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:67 +0x8a fp=0xc0531ebfe0 sp=0xc0531ebf88 pc=0x11d734a runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0531ebfe8 sp=0xc0531ebfe0 pc=0x484321 created by github.com/panjf2000/ants/v2.(goWorker).run in goroutine 3278846 /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:48 +0x5c

nightly-regression-dis-tp-cn-9spxv 重启log: cnlog.log

nightly-regression-dis-tp-cn-xp8xl重启log： nightly-regression-dis-tp-cn-xp8xl.log

企业微信截图_039eeaea-b00e-44ea-b646-5ce33cbe8501

Expected Behavior

No response

Steps to Reproduce

tke regression sysbench1000w cluster by table load data

Additional information

No response

matrixorigin / matrixone