matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 277 forks source link

[Bug]: [date 11.21]main tke regression: dn reported panic error invalid memory address or nil pointer dereference #20272

Open heni02 opened 10 hours ago

heni02 commented 10 hours ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

914d89b49

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11955458269/job/33337178111 image panic时间点:2024-11-21 19:23:41 (日志时间+8h)

企业微信截图_1ce1a183-e590-4f4a-8d76-ced8324aa13a

cn和dn panic error基本在同时间段报错 dn panic error: dn_panic.log

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2c259ec]

goroutine 1636 gp=0xc0075b1dc0 m=18 mp=0xc001f80708 [running]:
panic({0x4710020?, 0x8353c30?})
    /usr/local/go/src/runtime/panic.go:804 +0x168 fp=0xc2700b4ba0 sp=0xc2700b4af0 pc=0x47bc88
runtime.panicmem(...)
    /usr/local/go/src/runtime/panic.go:262
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:900 +0x359 fp=0xc2700b4c00 sp=0xc2700b4ba0 pc=0x47e339
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.init.containsABlkFuncFactory[...].func34.1(0x10?, 0x0)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/functions.go:411 +0x2ac fp=0xc2700b4e38 sp=0xc2700b4c00 pc=0x2c259ec
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.ForeachWindowFixed[...](0xc0c76b3bc0, 0x0, 0x5, 0x0, 0xc142540ba0, 0x0, 0xc20b0cf170)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers/utils.go:904 +0x779 fp=0xc2700b4fa8 sp=0xc2700b4e38 pc=0x222f879
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.ForeachVectorWindow({0x58bf540, 0xc25a440000}, 0x0, 0x5, {0x46e2940, 0xc142540ba0}, 0x0, 0xc20b0cf170)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers/utils.go:736 +0xa05 fp=0xc2700b5038 sp=0xc2700b4fa8 pc=0x222a345
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.ForeachVector({0x58bf540, 0xc25a440000}, {0x46e2940, 0xc142540ba0}, 0xc20b0cf170)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers/utils.go:405 +0x59 fp=0xc2700b5088 sp=0xc2700b5038 pc=0x22298f9
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(*baseObject).containsWithLoad(0xc06686e780, {0x57ef278, 0xc00105b5f0}, {0x7f11281e44d0, 0xc2a6f7bd00}, {0x58bf540, 0xc25a440000}, 0xc20b0cf170, 0x0, 0x1, ...)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/base.go:346 +0x4b4 fp=0xc2700b5278 sp=0xc2700b5088 pc=0x2c3d274
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(*baseObject).persistedContains(0xc06686e780, {0x57ef278, 0xc00105b5f0}, {0x7f11281e44d0, 0xc2a6f7bd00}, 0x1, {0x58bf540, 0xc25a440000}, {0xc16e3a28c0, 0x40, ...}, ...)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/base.go:428 +0x39e fp=0xc2700b53e0 sp=0xc2700b5278 pc=0x2c3e0de
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(*aobject).Contains(0xc0844a3158, {0x57ef278, 0xc00105b5f0}, {0x7f11281e44d0, 0xc2a6f7bd00}, 0x1, {0x58bf540, 0xc25a440000}, {0xc16e3a28c0, 0x40, ...}, ...)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/aobj.go:294 +0x196 fp=0xc2700b54b8 sp=0xc2700b53e0 pc=0x2c39d16
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnTable).findDeletes(0xc12af47b80, {0x57ef278, 0xc00105b5f0}, {0x58bf540, 0xc25a440000}, 0x0, 0x1)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/table.go:1140 +0x4cf fp=0xc2700b5700 sp=0xc2700b54b8 pc=0x3c3e1ef
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnTable).DoPrecommitDedupByPK.func1()
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/table.go:1173 +0x126 fp=0xc2700b57e8 sp=0xc2700b5700 pc=0x3c3e546
runtime/trace.WithRegion({0x57ef098?, 0x8596860?}, {0x5054204, 0x21}, 0xc2700b58d0)
    /usr/local/go/src/runtime/trace/annotation.go:141 +0xd1 fp=0xc2700b5850 sp=0xc2700b57e8 pc=0xc05051
github.com/matrixorigin/matrixone/pkg/common/moprobe.WithArgAndFn({0x57ef098, 0x8596860}, 0x8, 0x0, 0x0, 0xc2700b58d0)
    /go/src/github.com/matrixorigin/matrixone/pkg/common/moprobe/probe.go:99 +0x85 fp=0xc2700b5890 sp=0xc2700b5850 pc=0x11db5c5
github.com/matrixorigin/matrixone/pkg/common/moprobe.WithRegion(...)
    /go/src/github.com/matrixorigin/matrixone/pkg/common/moprobe/probe.go:88
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnTable).DoPrecommitDedupByPK(0xc0e632dcc0?, {0x58bf540?, 0xc269203f60?}, {0x0?, 0x0?, 0x0?}, 0x28?)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/table.go:1165 +0x86 fp=0xc2700b5910 sp=0xc2700b5890 pc=0x3c3e3a6
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnTable).PrePrepareDedup(0xc12af47b80, {0x57ef278, 0xc25bb60510}, 0x0)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/table.go:1054 +0x2e7 fp=0xc2700b5ad0 sp=0xc2700b5910 pc=0x3c3d267
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnDB).PrePrepare(0xc194c260c0, {0x57ef278, 0xc25bb60510})
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/txndb.go:481 +0x236 fp=0xc2700b5bd8 sp=0xc2700b5ad0 pc=0x3c49816
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(*txnStore).PrePrepare(0xc2426c5970, {0x57ef278, 0xc25bb60510})
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl/store.go:743 +0xb7 fp=0xc2700b5cc0 sp=0xc2700b5bd8 pc=0x3c33597
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase.(*Txn).PrePrepare(0x8596860?, {0x57ef278?, 0xc25bb60510?})
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase/txn.go:379 +0x29 fp=0xc2700b5ce8 sp=0xc2700b5cc0 pc=0x28b45c9
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase.(*TxnManager).onPrePrepare(0xc0070625b0, 0xc22a413da0)
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase/txnmgr.go:294 +0x12d fp=0xc2700b5da0 sp=0xc2700b5ce8 pc=0x28b8c4d
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase.(*TxnManager).dequeuePreparing(0xc0070625b0, {0xc010238008, 0x3e8, 0x3e8})
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase/txnmgr.go:463 +0x212 fp=0xc2700b5eb8 sp=0xc2700b5da0 pc=0x28ba2f2
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnbase.(*TxnManager).dequeuePreparing-fm({0xc010238008?, 0xc2700b5f30?, 0x0?})
    <autogenerated>:1 +0x31 fp=0xc2700b5ee8 sp=0xc2700b5eb8 pc=0x28c3171
github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/sm.(*safeQueue).Start.func1()
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/sm/safeq.go:89 +0x1ea fp=0xc2700b5fe0 sp=0xc2700b5ee8 pc=0x28ab5ea
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc2700b5fe8 sp=0xc2700b5fe0 pc=0x484d41
created by github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/sm.(*safeQueue).Start in goroutine 391
    /go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/tae/logstore/sm/safeq.go:68 +0xe5

nightly-regression-dis-tp-cn-8jffv cn pod panic error: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22CbW%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-914d89b49-20241121%5C%22,%20pod%3D%5C%22nightly-regression-dis-tp-cn-8jffv%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

企业微信截图_96fffdce-2258-483d-8820-6f7de2be14b5

nightly-regression-dis-tp-cn-v4fcr cn pod panic error: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22CbW%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-914d89b49-20241121%5C%22,%20pod%3D%5C%22nightly-regression-dis-tp-cn-v4fcr%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221732113834238%22,%22to%22:%221732286634238%22%7D%7D%7D&schemaVersion=1&orgId=1 image

Expected Behavior

No response

Steps to Reproduce

main tke regression test

Additional information

No response

heni02 commented 10 hours ago

cn panic已知问题,issue:https://github.com/matrixorigin/matrixone/issues/20200

dn panic问题这个单跟踪