search

pingcap / tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.
Apache License 2.0
292 stars 131 forks source link

validate Config.SyncerConfig.To is a nullptr or not for fix "drainer panic: runtime error" #143

Closed hicqu closed 7 years ago

hicqu commented 7 years ago

launch command and arguments:

$ ~/.go/src/github.com/pingcap/tidb-binlog/bin/drainer --pd-urls=http://arch-dev:2379 --addr=arch-dev:8249 --data-dir=./drainer --L=debug

error logs and stdout:

2017/04/19 11:29:20 version.go:23: [info] Build TS: 2017-04-19 03:28:59
2017/04/19 11:29:20 version.go:24: [info] Go Version: go1.8.1
2017/04/19 11:29:20 version.go:25: [info] Go OS/Arch: linuxamd64
2017/04/19 11:29:20 client.go:97: [info] [pd] create pd client with endpoints [http://arch-dev:2379]
2017/04/19 11:29:20 client.go:179: [info] [pd] leader switches to: http://192.168.56.200:2379, previous:
2017/04/19 11:29:20 client.go:115: [info] [pd] init cluster id 6409853286206803787
2017/04/19 11:29:20 server.go:78: [info] clusterID of drainer server is 6409853286206803787
2017/04/19 11:29:20 client.go:273: [error] [pd] create tso stream error: rpc error: code = 1 desc = context canceled
2017/04/19 11:29:20 client.go:97: [info] [pd] create pd client with endpoints [http://arch-dev:2379]
2017/04/19 11:29:20 client.go:179: [info] [pd] leader switches to: http://192.168.56.200:2379, previous:
2017/04/19 11:29:20 client.go:115: [info] [pd] init cluster id 6409853286206803787
2017/04/19 11:29:20 client.go:97: [info] [pd] create pd client with endpoints [arch-dev:2379]
2017/04/19 11:29:20 client.go:179: [info] [pd] leader switches to: http://192.168.56.200:2379, previous:
2017/04/19 11:29:20 client.go:115: [info] [pd] init cluster id 6409853286206803787
2017/04/19 11:29:20 scan.go:132: [debug] txn getData nextStartKey["mDDLJobHi\xffstory\x00\x00\x00\xfc\x00\x00\x00\x00\x00\x00\x00h"], txn 391268941171523587
2017/04/19 11:29:30 collector.go:167: [info] node arch-dev:8250 get save point {0 0}
2017/04/19 11:29:40 schema.go:38: [info] [local schema/table] map[29:{test t1} 41:{test a} 44:{test b} 35:{test c}]
2017/04/19 11:29:40 schema.go:39: [info] [local schema] map[1:0xc4203e24d0]
2017/04/19 11:29:40 schema.go:40: [info] [ignore schema] map[7:{}]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xdf2919]

goroutine 168 [running]:
github.com/pingcap/tidb-binlog/drainer/executor.newMysql(0x0, 0xc420155b30, 0x44239b, 0x10, 0xf24220)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/executor/mysql.go:14 +0x29
github.com/pingcap/tidb-binlog/drainer/executor.New(0x1049008, 0x5, 0x0, 0xc42033f1a0, 0x0, 0x1, 0x0)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/executor/executor.go:19 +0x149
github.com/pingcap/tidb-binlog/drainer.createExecutors(0x1049008, 0x5, 0x0, 0x1, 0x1048aea, 0x5, 0x1047314, 0x2, 0x1047f9a)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/util.go:195 +0xab
github.com/pingcap/tidb-binlog/drainer.(*Syncer).run(0xc42025c240, 0xc420447f50, 0x18, 0x20)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:484 +0x9c
github.com/pingcap/tidb-binlog/drainer.(*Syncer).Start(0xc42025c240, 0xc420406000, 0x18, 0x20, 0x0, 0x0)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/syncer.go:86 +0xa7
github.com/pingcap/tidb-binlog/drainer.(*Server).StartSyncer.func1(0xc4201bb180, 0xc420406000, 0x18, 0x20)
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/server.go:175 +0x81
created by github.com/pingcap/tidb-binlog/drainer.(*Server).StartSyncer
        /home/qupeng/.go/src/github.com/pingcap/tidb-binlog/drainer/server.go:180 +0x7a
hicqu commented 7 years ago

The reason is we specify --dest-db-type=mysql, but no connection arguments are passed so that Config.SyncerConfig.To is a nullptr.

To fix it, we should pass host, port, username, password to drainer. I will add these command arguments:

--dest-mysql-host  # default "localhost", used when --dest-db-type=mysql
--dest-mysql-port  # default 3306
--dest-mysql-user  # default "root"
--dest-mysql-password  # default ""

--dest-pb-dir  # default "${data-dir}/pb", used when --dest-db-type=pb

@GregoryIan , is that OK?

IANTHEREAL commented 7 years ago

we had these parameters in the past, but I think there're too many arguments. How about just making up those argument in Parse()?

hicqu commented 7 years ago

I figure out that we must specify syncer.to section in some.toml, and pass --config some.toml to drainer. Without that, Config.SyncerConfig.To will be a nullptr so that panic happends.

The default arguments like dest-mysql-host doesn't matter here. So, I will just check Config.SyncerConfig.To is a nullptr or not in Config.validate().

IANTHEREAL commented 7 years ago

ok

IANTHEREAL commented 7 years ago

Now I think we should let drainer running although cfg.to is nullptr

hicqu commented 7 years ago

So we should add a default down-stream MySQL config? If so, we should also tell users what the default config is, show that in ./drainer --help?