[Bug]: [0321] sysbench update non-index test on tke report 'Lost connection to MySQL server during query'.

Ariznawlll commented 8 months ago

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch Name

main

Commit ID

71f3515c9270840cc040b276878c6c91993ceae2

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job url: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/8363373798/job/22913372584 (sysbench oltp test 100w->update-non-index 100 theads)

查看了出错(Lost connection to MySQL server during query)期间的内存使用情况，非oom： https://grafana.ci.matrixorigin.cn/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=prometheus&var-cluster=&var-namespace=mo-nightly-regression-20240320&from=1710991969000&to=1710992209000

pod状态：

mo日志：https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22mvG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240320%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

trigger daily regression on tke.

Additional information

No response

w-zr commented 8 months ago

sukki37 commented 8 months ago

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22mvG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240320%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221710992036000%22,%22to%22:%221710992042000%22%7D%7D%7D&schemaVersion=1&orgId=1

{"level":"FATAL","time":"2024/03/21 03:33:57.622511 +0000","caller":"disttae/filter.go:1066","msg":"object stats has zero rows, detail: [object stats]: objName: 018e5f12-0e60-7db6-8cdf-699963204316_00000; extent: 1_1652645_611_2327; rowCnt: 0; blkCnt: 1; sortKey zoneMap: ZM(ANY)0[,]--; size: 0; originSize: 0; entryState: true; sorted: false; hasDeltaLoc: true; commitTS: 1710992034937299620-0; createTS: 1710992002656196389-0; deleteTS: 1710992029901042664-0","stacktrace":"github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.ExecuteBlockFilter.func1\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/filter.go:1066\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.ForeachSnapshotObjects\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/util.go:1484\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.ExecuteBlockFilter\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/filter.go:1020\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.TryFastFilterBlocks\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/filter.go:987\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(txnTable).tryFastFilterBlocks\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn_table.go:892\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(txnTable).rangesOnePart\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn_table.go:642\ngithub.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(txnTable).Ranges\n\t/go/src/github.com/matrixorigin/matrixone/pkg/vm/engine/disttae/txn_table.go:598\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).expandRanges\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:3604\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).generateNodes\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:3708\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compileTableScan\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:2096\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compilePlanScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1164\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compilePlanScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1186\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compilePlanScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1186\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compilePlanScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1605\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compilePlanScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1717\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compileQuery\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:1004\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).compileScope\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:683\ngithub.com/matrixorigin/matrixone/pkg/sql/compile.(Compile).Compile\n\t/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:298\ngithub.com/matrixorigin/matrixone/pkg/frontend.(TxnComputationWrapper).Compile\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/computation_wrapper.go:349\ngithub.com/matrixorigin/matrixone/pkg/frontend.(MysqlCmdExecutor).executeStmt\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:3738\ngithub.com/matrixorigin/matrixone/pkg/frontend.(MysqlCmdExecutor).doComQuery\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:4371\ngithub.com/matrixorigin/matrixone/pkg/frontend.(MysqlCmdExecutor).ExecRequest\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:4552\ngithub.com/matrixorigin/matrixone/pkg/frontend.(Routine).handleRequest\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine.go:284\ngithub.com/matrixorigin/matrixone/pkg/frontend.(RoutineManager).Handler\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine_manager.go:542\ngithub.com/matrixorigin/matrixone/pkg/frontend.(MOServer).handleMessage\n\t/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:145\ngithub.com/fagongzi/goetty/v2.(server).doStart.func2.1\n\t/go/pkg/mod/github.com/matrixorigin/goetty/v2@v2.0.0-20231122095211-6a25dc9130ca/application.go:356"}