Closed xzxiong closed 1 year ago
repro steps:
startup mo
untar data files mo-data.statement.tgz tar zxfv mo-data.statement.tgz -C {folder_where_mo-service_startup}
create table and create view
create database test;
use test;
CREATE EXTERNAL TABLE IF NOT EXISTS \`statement_info\`(
\`statement_id\` varchar(36) DEFAULT "0" COMMENT "statement uniq id",
\`transaction_id\` varchar(36) DEFAULT "0" COMMENT "txn uniq id",
\`session_id\` varchar(36) DEFAULT "0" COMMENT "session uniq id",
\`account\` varchar(1024) NOT NULL COMMENT "account name",
\`user\` varchar(1024) NOT NULL COMMENT "user name",
\`host\` varchar(1024) NOT NULL COMMENT "user client ip",
\`database\` varchar(1024) NOT NULL COMMENT "what database current session stay in.",
\`statement\` TEXT NOT NULL COMMENT "sql statement",
\`statement_tag\` TEXT NOT NULL COMMENT "note tag in statement(Reserved)",
\`statement_fingerprint\` TEXT NOT NULL COMMENT "note tag in statement(Reserved)",
\`node_uuid\` varchar(36) DEFAULT "0" COMMENT "node uuid, which node gen this data.",
\`node_type\` varchar(64) DEFAULT "node" COMMENT "node type in MO, val in [DN, CN, LOG]",
\`request_at\` datetime(6) NOT NULL COMMENT "request accept datetime",
\`response_at\` datetime(6) NOT NULL COMMENT "response send datetime",
\`duration\` bigint unsigned DEFAULT "0" COMMENT "exec time, unit: ns",
\`status\` varchar(32) DEFAULT "Running" COMMENT "sql statement running, enum: Running, Success, Failed",
\`err_code\` varchar(1024) DEFAULT "0" COMMENT "",
\`error\` TEXT NOT NULL COMMENT "error message",
\`exec_plan\` JSON NOT NULL COMMENT "statement execution plan",
\`rows_read\` bigint unsigned DEFAULT "0" COMMENT "rows read total",
\`bytes_scan\` bigint unsigned DEFAULT "0" COMMENT "bytes scan total"
) infile { "filepath" = './mo-data1/etl/*/*/*/*/*/statement_info/*.csv',
"compression" = "none" } FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 0 lines
exec sql
select statement_id, transaction_id, session_id, user, statement, node_uuid, node_type, request_at, response_at, duration / 1e9 as sec from statement_info where statement = 'show databases' ;
Have tried many times, but cannot repro.
It is difficult to repro, and cannot be located from the code. recommend to set it to s1 for now.
new repro step
commit id: e258d94d6
dev: macbook m1
cmd: ./mo-service -debug-http 127.0.0.1:8123 -allocs-profile ./allocs.profile -launch ./etc/launch-tae-logservice/launch.toml
PS: Occasional
dev1:/mnt/vdb/xiezexiong/issue/issue/6552/repro.tool.tgz
tar zxfv repro.tool.tgz -C {folder_where_mo-service_startup}
cd {folder_where_mo-service_startup} && sh output-log.sh
new panic log, mo-service.1118.log
Cannot repro... but there are many other errors during repro with the output_log.sh script. I suggest fix other bugs first and then fix this one.
Please @xzxiong help to check these panics whether issues related with them are created.
let wait those related issue https://github.com/matrixorigin/matrixone/issues/6553 https://github.com/matrixorigin/matrixone/issues/6748
another dragonboat panic:
panic: reflect: call of reflect.Value.IsNil on uint64 Value [recovered]
panic: reflect: call of reflect.Value.IsNil on uint64 Value
goroutine 10 [running]:
encoding/gob.catchError(0x14011e3a090)
/usr/local/go/src/encoding/gob/error.go:38 +0x80
panic({0x1022142a0, 0x140183c0018})
/usr/local/go/src/runtime/panic.go:884 +0x204
reflect.Value.IsNil(...)
/usr/local/go/src/reflect/value.go:1554
encoding/gob.encOpFor.func3(0x102420ac0?, 0x1400eba6190?, {0x102175580?, 0x1400eba6190?, 0x1400f191368?})
/usr/local/go/src/encoding/gob/encode.go:567 +0x16c
encoding/gob.(*Encoder).encodeStruct(0x14011e3a000, 0x14011e3a038, 0x1400062e2e8, {0x102420ac0?, 0x1400eba6190?, 0x9?})
/usr/local/go/src/encoding/gob/encode.go:328 +0x244
encoding/gob.encOpFor.func4(0x102140120?, 0x1400b2921c0, {0x102420ac0?, 0x1400eba6190?, 0x14011e3a000?})
/usr/local/go/src/encoding/gob/encode.go:581 +0xbc
encoding/gob.(*Encoder).encodeArray(0x1400b292180?, 0x1400599c0f0?, {0x102140120?, 0x14011e38048?, 0x102317600?}, 0x1400061e2a0, 0x0, 0x2, 0x0)
/usr/local/go/src/encoding/gob/encode.go:351 +0x400
encoding/gob.encOpFor.func1(0x102317600?, 0x1400b292180, {0x102140120?, 0x14011e38048?, 0x1026c0858?})
/usr/local/go/src/encoding/gob/encode.go:551 +0x114
encoding/gob.(*Encoder).encodeStruct(0x14011e3a000, 0x14011e3a038, 0x1400062e2d0, {0x102317600?, 0x14011e38040?, 0x10029101c?})
/usr/local/go/src/encoding/gob/encode.go:328 +0x244
encoding/gob.(*Encoder).encode(0x14011e3a000, 0x14011e34150?, {0x102317600?, 0x14011e38040?, 0x14000636380?}, 0x14000636380)
/usr/local/go/src/encoding/gob/encode.go:701 +0x180
encoding/gob.(*Encoder).EncodeValue(0x14011e3a000, {0x102317600?, 0x14011e38040?, 0x102440aa0?})
/usr/local/go/src/encoding/gob/encoder.go:251 +0x3f8
encoding/gob.(*Encoder).Encode(0x102317600?, {0x102317600?, 0x14011e38040?})
/usr/local/go/src/encoding/gob/encoder.go:176 +0xd8
github.com/lni/dragonboat/v4/internal/registry.getCompressedData(0x1, {0x1400eba6190, 0x2?, 0x2}, 0x2)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/view.go:171 +0x304
github.com/lni/dragonboat/v4/internal/registry.(*view).getFullSyncData(0x1400481a600)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/view.go:187 +0x40
github.com/lni/dragonboat/v4/internal/registry.(*delegate).LocalState(0x1400480c580?, 0x22?)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:258 +0x68
github.com/hashicorp/memberlist.(*Memberlist).sendLocalState(0x14004864000, {0x10267f890, 0x1400bdf01b8}, 0x1, {0x0, 0x0})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/net.go:1013 +0x324
github.com/hashicorp/memberlist.(*Memberlist).sendAndReceiveState(0x14004864000, {{0x1400a85e020?, 0x3?}, {0x0?, 0x1400a85e010?}}, 0x9?)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/net.go:959 +0x200
github.com/hashicorp/memberlist.(*Memberlist).pushPullNode(0x14004864000, {{0x1400a85e020?, 0xf?}, {0x0?, 0x1400a69eda8?}}, 0xc?)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/state.go:659 +0xbc
github.com/hashicorp/memberlist.(*Memberlist).Join(0x14004864000, {0x14000034860?, 0x1, 0x140048640b0?})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/memberlist@v0.4.1-0.20221101065119-bb7bce164406/memberlist.go:281 +0x274
github.com/lni/dragonboat/v4/internal/registry.(*gossipManager).join(0x14004864000?, {0x14000034860?, 0x0?, 0x0?})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:369 +0x24
github.com/lni/dragonboat/v4/internal/registry.newGossipManager.func1()
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/dragonboat/v4@v4.0.0-20221116064434-34611fc96385/internal/registry/gossip.go:359 +0xe8
github.com/lni/goutils/syncutil.(*Stopper).runWorker.func1()
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:79 +0xb8
created by github.com/lni/goutils/syncutil.(*Stopper).runWorker
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/github.com/matrixorigin/goutils@v1.3.1-0.20220604063047-388d67b4dbc4/syncutil/stopper.go:74 +0xec
panic: reflect: call of reflect.Value.IsNil on uint64 Value [recovered]
panic: reflect: call of reflect.Value.IsNil on uint64 Value
goroutine 275 [running]:
encoding/json.(*encodeState).marshal.func1()
/usr/local/go/src/encoding/json/encode.go:327 +0x80
panic({0x1068b02a0, 0x14000729e90})
/usr/local/go/src/runtime/panic.go:884 +0x204
reflect.Value.IsNil(...)
/usr/local/go/src/reflect/value.go:1554
encoding/json.sliceEncoder.encode({0x14011d204d8?}, 0x104aade20?, {0x106811580?, 0x1401fc9a660?, 0xb?}, {0x9?, 0x0?})
/usr/local/go/src/encoding/json/encode.go:869 +0x44c
encoding/json.structEncoder.encode({{{0x14005dd8d80?, 0x104aae4f8?, 0x1400583a1a0?}, 0x14005de3830?}}, 0x1400ebfca80, {0x106b144c0?, 0x1401fc9a660?, 0x104aae504?}, {0x20?, 0x7f?})
/usr/local/go/src/encoding/json/encode.go:760 +0x178
encoding/json.mapEncoder.encode({0x14011d20848?}, 0x1400ebfca80, {0x1068d35a0?, 0x1400a89d9e0?, 0xfe0001400ebfca80?}, {0x9?, 0x0?})
/usr/local/go/src/encoding/json/encode.go:813 +0x48c
encoding/json.structEncoder.encode({{{0x14003179050?, 0x106d5c858?, 0x107ee69c0?}, 0x14005de3890?}}, 0x1400ebfca80, {0x106a8e860?, 0x1400a89d9e0?, 0x106a8e860?}, {0xf0?, 0xeb?})
/usr/local/go/src/encoding/json/encode.go:760 +0x178
encoding/json.(*encodeState).reflectValue(0x14011d20a08?, {0x106a8e860?, 0x1400a89d9e0?, 0x104c99b49?}, {0xc0?, 0xe9?})
/usr/local/go/src/encoding/json/encode.go:359 +0x70
encoding/json.(*encodeState).marshal(0x600000101013fe0?, {0x106a8e860?, 0x1400a89d9e0?}, {0x0?, 0x6?})
/usr/local/go/src/encoding/json/encode.go:331 +0x120
encoding/json.(*Encoder).Encode(0x14005c319a0, {0x106a8e860, 0x1400a89d9e0})
/usr/local/go/src/encoding/json/stream.go:206 +0x58
go.uber.org/zap/zapcore.(*jsonEncoder).encodeReflected(0x1400b88e9c0, {0x106a8e860, 0x1400a89d9e0})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/json_encoder.go:174 +0x54
go.uber.org/zap/zapcore.(*jsonEncoder).AddReflected(0x1400b88e9c0, {0x106349a6f, 0x8}, {0x106a8e860?, 0x1400a89d9e0?})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/json_encoder.go:182 +0x38
go.uber.org/zap/zapcore.Field.AddTo({{0x106349a6f, 0x8}, 0x17, 0x0, {0x0, 0x0}, {0x106a8e860, 0x1400a89d9e0}}, {0x106d566e0, 0x1400b88e9c0?})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:170 +0x6d4
go.uber.org/zap/zapcore.addFields(...)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/field.go:210
go.uber.org/zap/zapcore.consoleEncoder.writeContext({0x106ca8540?}, 0x14007ef7500, {0x1401316e500, 0x5, 0x1?})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:141 +0xd8
go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry({0x106810700?}, {0xff, {0xc0d783769c325688, 0x1712d0e280, 0x107eb3740}, {0x106349b6f, 0x8}, {0x106360cb8, 0x16}, {0x1, ...}, ...}, ...)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/console_encoder.go:119 +0x5bc
go.uber.org/zap/zapcore.(*ioCore).Write(0x140032d3200, {0xff, {0xc0d783769c325688, 0x1712d0e280, 0x107eb3740}, {0x106349b6f, 0x8}, {0x106360cb8, 0x16}, {0x1, ...}, ...}, ...)
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/core.go:86 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0x1400c938540, {0x1401316e500, 0x5, 0x5})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/zapcore/entry.go:220 +0x150
go.uber.org/zap.(*Logger).Debug(0x106aff9c0?, {0x106360cb8?, 0x14011d213a0?}, {0x1401316e500, 0x5, 0x5})
/Users/liubo/Workspace/Projects/GOPATH/pkg/mod/go.uber.org/zap@v1.21.0/logger.go:184 +0x54
github.com/matrixorigin/matrixone/pkg/hakeeper/checkers.(*Coordinator).Check(_, {_, _}, {0x3cb, {{0x1400a89cf30, 0x1, 0x1}, {0x1401fc9a3c0, 0x2, 0x2}, ...}, ...})
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/hakeeper/checkers/coordinator.go:59 +0x4c4
github.com/matrixorigin/matrixone/pkg/logservice.(*store).getScheduleCommand(0x14005566280, 0x1, 0x14011d21a78?, 0x1400d9a7a40)
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:291 +0xd8
github.com/matrixorigin/matrixone/pkg/logservice.(*store).healthCheck(0x14005566280, 0x1049696f8?, 0x14000728d08?)
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:170 +0x78
github.com/matrixorigin/matrixone/pkg/logservice.(*store).hakeeperCheck(0x14005566280)
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store_hakeeper_check.go:142 +0x28c
github.com/matrixorigin/matrixone/pkg/logservice.(*store).ticker(0x14005566280, {0x106ce4c50, 0x140000f6400})
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store.go:668 +0x288
github.com/matrixorigin/matrixone/pkg/logservice.(*store).startHAKeeperReplica.func1({0x106ce4c50, 0x140000f6400})
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/logservice/store.go:224 +0x5c
github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:258 +0x6c
created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask
/Users/liubo/Workspace/Projects/GOPATH/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:253 +0x150
This problem is most likely the same as #6553.
still working on it.
fix by #6980 in branch 0.6.0 and #6975 in branch main.
got it.
mark check.
after 12 hour run, exec output_log.sh
each 5min, mo is still runnning fine.
Is there an existing issue for the same bug?
Environment
Actual Behavior
select statement_id, transaction_id, session_id, user, statement, node_uuid, node_type, request_at, response_at, duration / 1e9 as sec from statement_info where statement = 'show databases' ;
ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query
last log output
Expected Behavior
No response
Steps to Reproduce
No response
Additional information
No response