search

matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.77k stars 274 forks source link

[Bug]: mo_cdc: cn restart reported panic error #19002

Open heni02 opened 6 days ago

heni02 commented 6 days ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

a77a735

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

minio 3cn env cn restart reported panic error，panic log: {"level":"ERROR","time":"2024/09/24 04:03:25.745934 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"ERROR","time":"2024/09/24 04:03:25.746352 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"ERROR","time":"2024/09/24 04:03:25.746631 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"ERROR","time":"2024/09/24 04:03:25.746760 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"ERROR","time":"2024/09/24 04:03:25.747120 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"ERROR","time":"2024/09/24 04:03:25.747325 +0000","caller":"cdc/sinker.go:596","msg":"----mysql send sql----, success"} {"level":"INFO","time":"2024/09/24 04:03:25.752663 +0000","caller":"cdc/watermark_updater.go:72","msg":"^^^^^ WatermarkUpdater.Run: end"} panic: runtime error: invalid memory address or nil pointer dereference goroutine 3010924 gp=0xc4ced02fc0 m=34 mp=0xc046e29c08 [running]: runtime.sigpanic() /go/src/github.com/matrixorigin/matrixone/pkg/common/log/logger.go:92 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtInBack(0xc1898bc008, 0xc188aa1b80) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2560 +0x325 fp=0xc0068db7a0 sp=0xc0068db6d8 pc=0x392eb25 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt({0x567e5f0, 0xc1898bc008}, 0xc188aa1b80, {0x564f8f0, 0xc1898bcc08}) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x2f1 fp=0xc0068dc838 sp=0xc0068db7a0 pc=0x392e4d1 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x567e5f0, 0xc1898bc008}, 0xc188aa1b80) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x605 fp=0xc0068ddae0 sp=0xc0068dc838 pc=0x392dc85 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x567e5f0, 0xc1898bc008}, 0xc188aa1b80) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2374 +0x109 fp=0xc0068ddb40 sp=0xc0068ddae0 pc=0x392d5a9 github.com/matrixorigin/matrixone/pkg/frontend.doComQueryInBack(0xc1898bc008, 0xc188aa1b80, 0xc1898b5080) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/back_exec.go:361 +0x15ad fp=0xc0068de058 sp=0xc0068ddb40 pc=0x38ecd8d github.com/matrixorigin/matrixone/pkg/frontend.(backExec).Exec(0xc08c20f7a8, {0x558d9d8, 0xc1898365a0}, {0xc1810c7ef0, 0xe2}) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/back_exec.go:127 +0x60b fp=0xc0068de1e8 sp=0xc0068de058 pc=0x38ea70b /go/src/github.com/matrixorigin/matrixone/pkg/frontend/authenticate.go:6599 +0xb1 fp=0xc0068de818 sp=0xc0068de7c0 pc=0x38d6571 /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2142 +0x13d fp=0xc0068de8a8 sp=0xc0068de818 pc=0x392a7bd /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:1951 +0x8af fp=0xc0068dea30 sp=0xc0068de8a8 pc=0x392916f github.com/matrixorigin/matrixone/pkg/frontend.(TxnComputationWrapper).Compile(0xc18982ab00, {0x44060e0?, 0xc189838140}, 0xc013b42198) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/computation_wrapper.go:192 +0x505 fp=0xc0068dec30 sp=0xc0068dea30 pc=0x390a065 github.com/matrixorigin/matrixone/pkg/frontend.executeStmt(0xc18983a008, 0xc189838140) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2655 +0xb0f fp=0xc0068deed8 sp=0xc0068dec30 pc=0x392f7cf github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt({0x567e308, 0xc18983a008}, 0xc189838140) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2558 +0x2c9 fp=0xc0068defa0 sp=0xc0068deed8 pc=0x392eac9 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt({0x567e308, 0xc18983a008}, 0xc189838140, {0x564f8f0, 0xc189840008}) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x2f1 fp=0xc0068e0038 sp=0xc0068defa0 pc=0x392e4d1 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace({0x567e308, 0xc18983a008}, 0xc189838140) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x605 fp=0xc0068e12e0 sp=0xc0068e0038 pc=0x392dc85 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn({0x567e308, 0xc18983a008}, 0xc189838140) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2374 +0x109 fp=0xc0068e1340 sp=0xc0068e12e0 pc=0x392d5a9 github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse(0xc18983a008, 0xc189838140) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2338 +0x3b3 fp=0xc0068e15e0 sp=0xc0068e1340 pc=0x392c8b3 github.com/matrixorigin/matrixone/pkg/frontend.doComQuery(0xc18983a008, 0xc189838140, 0xc18965b9e0) /go/src/github.com/matrixorigin/matrixone/pkg/frontend/internal_executor.go:159 +0x365 fp=0xc0068e1cf8 sp=0xc0068e1bf0 pc=0x3916625 github.com/matrixorigin/matrixone/pkg/cdc.(WatermarkUpdater).updateDb(0xc084fe8840, 0x428dd, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}) /go/src/github.com/matrixorigin/matrixone/pkg/cdc/watermark_updater.go:167 +0x193 fp=0xc0068e1db0 sp=0xc0068e1cf8 pc=0x386b6d3 /go/src/github.com/matrixorigin/matrixone/pkg/cdc/watermark_updater.go:157 +0xad fp=0xc0068e1e58 sp=0xc0068e1db0 pc=0x386a5ad sync.(Map).Range(0xc086eadf48?, 0xc0068e1f88) /usr/local/go/src/sync/map.go:501 +0x1f8 fp=0xc0068e1ef8 sp=0xc0068e1e58 pc=0x490e98 github.com/matrixorigin/matrixone/pkg/cdc.(WatermarkUpdater).flushAll(...) /go/src/github.com/matrixorigin/matrixone/pkg/cdc/watermark_updater.go:154 github.com/matrixorigin/matrixone/pkg/cdc.(WatermarkUpdater).Run(0xc084fe8840, {0x558d9d8, 0xc7857bdec0}, 0xc641f14bb0) /go/src/github.com/matrixorigin/matrixone/pkg/cdc/watermark_updater.go:83 +0xcc fp=0xc0068e1fb0 sp=0xc0068e1ef8 pc=0x386a3cc github.com/matrixorigin/matrixone/pkg/frontend.(CdcTask).startWatermarkAndPipeline.gowrap1() /go/src/github.com/matrixorigin/matrixone/pkg/frontend/cdc.go:1103 +0x2c fp=0xc0068e1fe0 sp=0xc0068e1fb0 pc=0x38f938c runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0068e1fe8 sp=0xc0068e1fe0 pc=0x484321 created by github.com/matrixorigin/matrixone/pkg/frontend.(CdcTask).startWatermarkAndPipeline in goroutine 3010717 /go/src/github.com/matrixorigin/matrixone/pkg/frontend/cdc.go:1103 +0x537

goroutine 1 gp=0xc0000081c0 m=nil [select, 1185 minutes]: runtime.gopark(0xc0035fde08?, 0x2?, 0xb0?, 0xc1?, 0xc0035fdd24?) /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc01895fbc0 sp=0xc01895fba0 pc=0x47b64e runtime.selectgo(0xc01895fe08, 0xc0035fdd20, 0x2?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc01895fce8 sp=0xc01895fbc0 pc=0x457005 main.waitSignalToStop(0xc0006e5000, 0xc000282bd0) /go/src/github.com/matrixorigin/matrixone/cmd/mo-service/main.go:135 +0x13e fp=0xc01895fe68 sp=0xc01895fce8 pc=0x3adb97e main.main() /go/src/github.com/matrixorigin/matrixone/cmd/mo-service/main.go:124 +0x2a8 fp=0xc01895ff50 sp=0xc01895fe68 pc=0x3adb5c8 runtime.main() /usr/local/go/src/runtime/proc.go:272 +0x28b fp=0xc01895ffe0 sp=0xc01895ff50 pc=0x444f0b runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc01895ffe8 sp=0xc01895ffe0 pc=0x484321

goroutine 2 gp=0xc000008c40 m=nil [force gc (idle), 7 minutes]: runtime.gopark(0x1ee203fd3d6038?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0001cafa8 sp=0xc0001caf88 pc=0x47b64e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:430 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:337 +0xb3 fp=0xc0001cafe0 sp=0xc0001cafa8 pc=0x445253 runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001cafe8 sp=0xc0001cafe0 pc=0x484321 created by runtime.init.7 in goroutine 1 /usr/local/go/src/runtime/proc.go:325 +0x1a

panic log: cn_panic.tar.gz

看日志时间应该是创建同步任务时报的错误，可能有关系

企业微信截图_3e1c165f-7bb3-464e-bf6e-7b5b26c7aa38

Expected Behavior

No response

Steps to Reproduce

./mo_cdc task create --task-name "cdc_tpcc" --source-uri="mysql://dump:111@10.222.6.6:6001" --sink-type="mysql" --sink-uri="mysql://dump:111@10.222.1.129:3306"    --tables='tpcc_100.bmsql_config:test_cdc_db.bmsql_config,tpcc_100.bmsql_customer:test_cdc_db.bmsql_customer,tpcc_100.bmsql_district:test_cdc_db.bmsql_district,tpcc_100.bmsql_history:test_cdc_db.bmsql_history,tpcc_100.bmsql_item:test_cdc_db.bmsql_item,tpcc_100.bmsql_new_order:test_cdc_db.bmsql_new_order,tpcc_100.bmsql_oorder:test_cdc_db.bmsql_oorder,tpcc_100.bmsql_order_line:test_cdc_db.bmsql_order_line,tpcc_100.bmsql_stock:test_cdc_db.bmsql_stock,tpcc_100.bmsql_warehouse:test_cdc_db.bmsql_warehouse' --level="account"  --account="sys"

Additional information

No response

heni02 commented 6 days ago

相关联的issue：#18964

ck89119 commented 4 days ago

session中logger没有初试化导致的空指针

处理中