matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.77k stars 274 forks source link

[Bug]: select from system.log_info meet panic: runtime error: integer divide by zero #6552

Closed xzxiong closed 1 year ago

xzxiong commented 1 year ago

Is there an existing issue for the same bug?

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93): 2dd39113e
- Hardware parameters:
- OS type:
- Others: `./mo-service -debug-http 127.0.0.1:8123 -allocs-profile ./allocs.profile -launch ./etc/launch-tae-logservice/launch.toml`

Actual Behavior

  1. startup mo
  2. exec select statement_id, transaction_id, session_id, user, statement, node_uuid, node_type, request_at, response_at, duration / 1e9 as sec from statement_info where statement = 'show databases' ;
    • got ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query

last log output

panic: runtime error: integer divide by zero

goroutine 165 [running]:
github.com/lni/dragonboat/v4/internal/server.(*FixedPartitioner).GetPartitionID(0x0?, 0x0?)
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/internal/server/partition.go:39 +0x38
github.com/lni/dragonboat/v4.(*workReady).shardReadyByUpdates(0x140007a64b0, {0x1400a880d80, 0x1, 0x10521f900?})
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:164 +0x180
github.com/lni/dragonboat/v4.(*engine).setApplyReadyByUpdates(...)
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:1443
github.com/lni/dragonboat/v4.(*engine).applySnapshotAndUpdate(0x14005500000?, {0x1400a880d80?, 0x1?, 0x1?}, 0x17c8b8?, 0x1?)
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:1398 +0x5c
github.com/lni/dragonboat/v4.(*engine).processSteps(0x14005500000, 0x14006e17d90?, 0x14006e17e30?, 0x1400550c7e0, {0x1075f0a50, 0x1?, 0x0}, 0x140007a01e0?)
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:1328 +0x2d8
github.com/lni/dragonboat/v4.(*engine).stepWorkerMain(0x14005500000, 0x1)
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:1254 +0x2bc
github.com/lni/dragonboat/v4.newExecEngine.func1()
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221031100441-62f1755ca7ab/engine.go:1047 +0x5c
github.com/lni/goutils/syncutil.(*Stopper).runWorker.func1()
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:79 +0xb8
created by github.com/lni/goutils/syncutil.(*Stopper).runWorker
    /Users/jacksonxie/go/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:74 +0xec

Expected Behavior

No response

Steps to Reproduce

No response

Additional information

No response

xzxiong commented 1 year ago

repro steps:

  1. startup mo

  2. untar data files mo-data.statement.tgz tar zxfv mo-data.statement.tgz -C {folder_where_mo-service_startup}

  3. create table and create view

    create database test;
    use test;
    CREATE EXTERNAL TABLE IF NOT EXISTS \`statement_info\`(
    \`statement_id\` varchar(36) DEFAULT "0" COMMENT "statement uniq id",
    \`transaction_id\` varchar(36) DEFAULT "0" COMMENT "txn uniq id",
    \`session_id\` varchar(36) DEFAULT "0" COMMENT "session uniq id",
    \`account\` varchar(1024) NOT NULL COMMENT "account name",
    \`user\` varchar(1024) NOT NULL COMMENT "user name",
    \`host\` varchar(1024) NOT NULL COMMENT "user client ip",
    \`database\` varchar(1024) NOT NULL COMMENT "what database current session stay in.",
    \`statement\` TEXT NOT NULL COMMENT "sql statement",
    \`statement_tag\` TEXT NOT NULL COMMENT "note tag in statement(Reserved)",
    \`statement_fingerprint\` TEXT NOT NULL COMMENT "note tag in statement(Reserved)",
    \`node_uuid\` varchar(36) DEFAULT "0" COMMENT "node uuid, which node gen this data.",
    \`node_type\` varchar(64) DEFAULT "node" COMMENT "node type in MO, val in [DN, CN, LOG]",
    \`request_at\` datetime(6) NOT NULL COMMENT "request accept datetime",
    \`response_at\` datetime(6) NOT NULL COMMENT "response send datetime",
    \`duration\` bigint unsigned DEFAULT "0" COMMENT "exec time, unit: ns",
    \`status\` varchar(32) DEFAULT "Running" COMMENT "sql statement running, enum: Running, Success, Failed",
    \`err_code\` varchar(1024) DEFAULT "0" COMMENT "",
    \`error\` TEXT NOT NULL COMMENT "error message",
    \`exec_plan\` JSON NOT NULL COMMENT "statement execution plan",
    \`rows_read\` bigint unsigned DEFAULT "0" COMMENT "rows read total",
    \`bytes_scan\` bigint unsigned DEFAULT "0" COMMENT "bytes scan total"
    ) infile { "filepath" = './mo-data1/etl/*/*/*/*/*/statement_info/*.csv',
    "compression" = "none" } FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 0 lines
  4. exec sql

    select statement_id, transaction_id, session_id, user, statement, node_uuid, node_type, request_at, response_at, duration / 1e9 as sec from statement_info where statement = 'show databases' ;
volgariver6 commented 1 year ago

Have tried many times, but cannot repro.

volgariver6 commented 1 year ago

It is difficult to repro, and cannot be located from the code. recommend to set it to s1 for now.

xzxiong commented 1 year ago

new repro step

commit id: e258d94d6 dev: macbook m1 cmd: ./mo-service -debug-http 127.0.0.1:8123 -allocs-profile ./allocs.profile -launch ./etc/launch-tae-logservice/launch.toml

PS: Occasional

  1. cleanup all datadir
  2. startup new mo
  3. import local etl data, with tool dev1:/mnt/vdb/xiezexiong/issue/issue/6552/repro.tool.tgz
    tar zxfv repro.tool.tgz -C {folder_where_mo-service_startup}
    cd  {folder_where_mo-service_startup} && sh output-log.sh
    企业微信截图_808f48c2-c09a-45d6-9d16-89ed9c6fe076

new panic log, mo-service.1118.log

企业微信截图_b4b0815e-c32f-465f-bec8-5122c88b78b6
volgariver6 commented 1 year ago

Cannot repro... but there are many other errors during repro with the output_log.sh script. I suggest fix other bugs first and then fix this one.

Please @xzxiong help to check these panics whether issues related with them are created.

WeChatWorkScreenshot_de3e13ea-9d8c-45e5-93e7-73d0b6049ea5

WeChatWorkScreenshot_7d7d061a-3bf4-4067-be72-aa67b4339673

WeChatWorkScreenshot_6a991e5d-571a-4219-89c3-0f46fcbf6f4e

xzxiong commented 1 year ago

let wait those related issue https://github.com/matrixorigin/matrixone/issues/6553 https://github.com/matrixorigin/matrixone/issues/6748

volgariver6 commented 1 year ago

another dragonboat panic:

panic: reflect: call of reflect.Value.IsNil on uint64 Value [recovered]
    panic: reflect: call of reflect.Value.IsNil on uint64 Value

goroutine 10 [running]:
encoding/gob.catchError(0x14011e3a090)
    /usr/local/go/src/encoding/gob/error.go:38 +0x80
panic({0x1022142a0, 0x140183c0018})
    /usr/local/go/src/runtime/panic.go:884 +0x204
reflect.Value.IsNil(...)
    /usr/local/go/src/reflect/value.go:1554
encoding/gob.encOpFor.func3(0x102420ac0?, 0x1400eba6190?, {0x102175580?, 0x1400eba6190?, 0x1400f191368?})
    /usr/local/go/src/encoding/gob/encode.go:567 +0x16c
encoding/gob.(*Encoder).encodeStruct(0x14011e3a000, 0x14011e3a038, 0x1400062e2e8, {0x102420ac0?, 0x1400eba6190?, 0x9?})
    /usr/local/go/src/encoding/gob/encode.go:328 +0x244
encoding/gob.encOpFor.func4(0x102140120?, 0x1400b2921c0, {0x102420ac0?, 0x1400eba6190?, 0x14011e3a000?})
    /usr/local/go/src/encoding/gob/encode.go:581 +0xbc
encoding/gob.(*Encoder).encodeArray(0x1400b292180?, 0x1400599c0f0?, {0x102140120?, 0x14011e38048?, 0x102317600?}, 0x1400061e2a0, 0x0, 0x2, 0x0)
    /usr/local/go/src/encoding/gob/encode.go:351 +0x400
encoding/gob.encOpFor.func1(0x102317600?, 0x1400b292180, {0x102140120?, 0x14011e38048?, 0x1026c0858?})
    /usr/local/go/src/encoding/gob/encode.go:551 +0x114
encoding/gob.(*Encoder).encodeStruct(0x14011e3a000, 0x14011e3a038, 0x1400062e2d0, {0x102317600?, 0x14011e38040?, 0x10029101c?})
    /usr/local/go/src/encoding/gob/encode.go:328 +0x244
encoding/gob.(*Encoder).encode(0x14011e3a000, 0x14011e34150?, {0x102317600?, 0x14011e38040?, 0x14000636380?}, 0x14000636380)
    /usr/local/go/src/encoding/gob/encode.go:701 +0x180
encoding/gob.(*Encoder).EncodeValue(0x14011e3a000, {0x102317600?, 0x14011e38040?, 0x102440aa0?})
    /usr/local/go/src/encoding/gob/encoder.go:251 +0x3f8
encoding/gob.(*Encoder).Encode(0x102317600?, {0x102317600?, 0x14011e38040?})
    /usr/local/go/src/encoding/gob/encoder.go:176 +0xd8
github.com/lni/dragonboat/v4/internal/registry.getCompressedData(0x1, {0x1400eba6190, 0x2?, 0x2}, 0x2)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/view.go:171 +0x304
github.com/lni/dragonboat/v4/internal/registry.(*view).getFullSyncData(0x1400481a600)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/view.go:187 +0x40
github.com/lni/dragonboat/v4/internal/registry.(*delegate).LocalState(0x1400480c580?, 0x22?)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:258 +0x68
github.com/hashicorp/memberlist.(*Memberlist).sendLocalState(0x14004864000, {0x10267f890, 0x1400bdf01b8}, 0x1, {0x0, 0x0})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/net.go:1013 +0x324
github.com/hashicorp/memberlist.(*Memberlist).sendAndReceiveState(0x14004864000, {{0x1400a85e020?, 0x3?}, {0x0?, 0x1400a85e010?}}, 0x9?)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/net.go:959 +0x200
github.com/hashicorp/memberlist.(*Memberlist).pushPullNode(0x14004864000, {{0x1400a85e020?, 0xf?}, {0x0?, 0x1400a69eda8?}}, 0xc?)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/state.go:659 +0xbc
github.com/hashicorp/memberlist.(*Memberlist).Join(0x14004864000, {0x14000034860?, 0x1, 0x140048640b0?})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/memberlist.go:281 +0x274
github.com/lni/dragonboat/v4/internal/registry.(*gossipManager).join(0x14004864000?, {0x14000034860?, 0x0?, 0x0?})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:369 +0x24
github.com/lni/dragonboat/v4/internal/registry.newGossipManager.func1()
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:359 +0xe8
github.com/lni/goutils/syncutil.(*Stopper).runWorker.func1()
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:79 +0xb8
created by github.com/lni/goutils/syncutil.(*Stopper).runWorker
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:74 +0xec
volgariver6 commented 1 year ago
panic: reflect: call of reflect.Value.IsNil on uint64 Value [recovered]
    panic: reflect: call of reflect.Value.IsNil on uint64 Value

goroutine 275 [running]:
encoding/json.(*encodeState).marshal.func1()
    /usr/local/go/src/encoding/json/encode.go:327 +0x80
panic({0x1068b02a0, 0x14000729e90})
    /usr/local/go/src/runtime/panic.go:884 +0x204
reflect.Value.IsNil(...)
    /usr/local/go/src/reflect/value.go:1554
encoding/json.sliceEncoder.encode({0x14011d204d8?}, 0x104aade20?, {0x106811580?, 0x1401fc9a660?, 0xb?}, {0x9?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:869 +0x44c
encoding/json.structEncoder.encode({{{0x14005dd8d80?, 0x104aae4f8?, 0x1400583a1a0?}, 0x14005de3830?}}, 0x1400ebfca80, {0x106b144c0?, 0x1401fc9a660?, 0x104aae504?}, {0x20?, 0x7f?})
    /usr/local/go/src/encoding/json/encode.go:760 +0x178
encoding/json.mapEncoder.encode({0x14011d20848?}, 0x1400ebfca80, {0x1068d35a0?, 0x1400a89d9e0?, 0xfe0001400ebfca80?}, {0x9?, 0x0?})
    /usr/local/go/src/encoding/json/encode.go:813 +0x48c
encoding/json.structEncoder.encode({{{0x14003179050?, 0x106d5c858?, 0x107ee69c0?}, 0x14005de3890?}}, 0x1400ebfca80, {0x106a8e860?, 0x1400a89d9e0?, 0x106a8e860?}, {0xf0?, 0xeb?})
    /usr/local/go/src/encoding/json/encode.go:760 +0x178
encoding/json.(*encodeState).reflectValue(0x14011d20a08?, {0x106a8e860?, 0x1400a89d9e0?, 0x104c99b49?}, {0xc0?, 0xe9?})
    /usr/local/go/src/encoding/json/encode.go:359 +0x70
encoding/json.(*encodeState).marshal(0x600000101013fe0?, {0x106a8e860?, 0x1400a89d9e0?}, {0x0?, 0x6?})
    /usr/local/go/src/encoding/json/encode.go:331 +0x120
encoding/json.(*Encoder).Encode(0x14005c319a0, {0x106a8e860, 0x1400a89d9e0})
    /usr/local/go/src/encoding/json/stream.go:206 +0x58
go.uber.org/zap/zapcore.(*jsonEncoder).encodeReflected(0x1400b88e9c0, {0x106a8e860, 0x1400a89d9e0})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/json_encoder.go:174 +0x54
go.uber.org/zap/zapcore.(*jsonEncoder).AddReflected(0x1400b88e9c0, {0x106349a6f, 0x8}, {0x106a8e860?, 0x1400a89d9e0?})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/json_encoder.go:182 +0x38
go.uber.org/zap/zapcore.Field.AddTo({{0x106349a6f, 0x8}, 0x17, 0x0, {0x0, 0x0}, {0x106a8e860, 0x1400a89d9e0}}, {0x106d566e0, 0x1400b88e9c0?})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:170 +0x6d4
go.uber.org/zap/zapcore.addFields(...)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:210
go.uber.org/zap/zapcore.consoleEncoder.writeContext({0x106ca8540?}, 0x14007ef7500, {0x1401316e500, 0x5, 0x1?})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:141 +0xd8
go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry({0x106810700?}, {0xff, {0xc0d783769c325688, 0x1712d0e280, 0x107eb3740}, {0x106349b6f, 0x8}, {0x106360cb8, 0x16}, {0x1, ...}, ...}, ...)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:119 +0x5bc
go.uber.org/zap/zapcore.(*ioCore).Write(0x140032d3200, {0xff, {0xc0d783769c325688, 0x1712d0e280, 0x107eb3740}, {0x106349b6f, 0x8}, {0x106360cb8, 0x16}, {0x1, ...}, ...}, ...)
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/core.go:86 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x1400c938540, {0x1401316e500, 0x5, 0x5})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/entry.go:220 +0x150
go.uber.org/zap.(*Logger).Debug(0x106aff9c0?, {0x106360cb8?, 0x14011d213a0?}, {0x1401316e500, 0x5, 0x5})
    /Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/logger.go:184 +0x54
github.com/matrixorigin/matrixone/pkg/hakeeper/checkers.(*Coordinator).Check(_, {_, _}, {0x3cb, {{0x1400a89cf30, 0x1, 0x1}, {0x1401fc9a3c0, 0x2, 0x2}, ...}, ...})
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/hakeeper/checkers/coordinator.go:59 +0x4c4
github.com/matrixorigin/matrixone/pkg/logservice.(*store).getScheduleCommand(0x14005566280, 0x1, 0x14011d21a78?, 0x1400d9a7a40)
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:291 +0xd8
github.com/matrixorigin/matrixone/pkg/logservice.(*store).healthCheck(0x14005566280, 0x1049696f8?, 0x14000728d08?)
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:170 +0x78
github.com/matrixorigin/matrixone/pkg/logservice.(*store).hakeeperCheck(0x14005566280)
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:142 +0x28c
github.com/matrixorigin/matrixone/pkg/logservice.(*store).ticker(0x14005566280, {0x106ce4c50, 0x140000f6400})
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store.go:668 +0x288
github.com/matrixorigin/matrixone/pkg/logservice.(*store).startHAKeeperReplica.func1({0x106ce4c50, 0x140000f6400})
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store.go:224 +0x5c
github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:258 +0x6c
created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask
    /Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:253 +0x150
volgariver6 commented 1 year ago

This problem is most likely the same as #6553.

volgariver6 commented 1 year ago

still working on it.

volgariver6 commented 1 year ago

fix by #6980 in branch 0.6.0 and #6975 in branch main.

xzxiong commented 1 year ago

got it.

xzxiong commented 1 year ago

mark check. after 12 hour run, exec output_log.sh each 5min, mo is still runnning fine.