apache / kvrocks-controller

Apache Kvrocks Controller is a cluster management tool for Apache Kvrocks.
https://kvrocks.apache.org/
Apache License 2.0
78 stars 42 forks source link

kvrocks_controller failover lead one controller panic #113

Closed bbqccx closed 10 months ago

bbqccx commented 10 months ago

Initial State: Control 1 is not running. Control 2 is running.

Operations: 1.Start Control 1 as a follower. 2.Kill Control 2 after a few hours. It was found that Control 1 did not take over and crashed . 3.Restart Control 1 as the leader and start Control 1 as a follower 4.Retry killing Control 2, and it was found that Control 1 could take over this time

Control1 log:

panic: close of closed channel goroutine 1 [running]: github.com/RocksLabs/kvrocks_controller/controller/migrate.(Migrator).Shutdown(0xc000124f00) /home/k v rock s/controller/kvrocks_controller/controller/migrate/migrator.go:79 +0x25 github.com/RocksLabs/kvrocks_controller/controller.(Controller).unloadModules(0xc000124f50) /home/kvrocks/controller/kvrocks_controller/controller/controller.go:92 +0x45 github.com/RocksLabs/kvrocks_controller/controller.(Controller).Stop.func1() /home/kvrocks/controller/kvrocks_controller/controller/controller.go:181 +0xcf sync.(Once).doSlow(0xc0004a6a20?, 0xc000044238?) /usr/local/go/src/sync/once.go:74 +0xc2 sync.(Once).Do(...) /usr/local/go/src/sync/once.go:65 github.com/RocksLabs/kvrocks_controller/controller.(Controller).Stop(0xbf6085?) /home/kvrocks/controller/kvrocks_controller/controller/controller.go:175 +0x45 github.com/RocksLabs/kvrocks_controller/server.(*Server).Stop(0xc00033e4b0) /home/kvrocks/controller/kvrocks_controller/server/server.go:115 +0x45 main.main() /home/kvrocks/controller/kvrocks_controller/cmd/server/main.go:101 +0x6c5 nohup: ignoring input {"level":"info","ts":1693818282.8894243,"caller":"server/main.go:75","msg":"Kvrocks controller is running with version: unstable"} {"level":"info","ts":1693818283.0661163,"caller":"controller/controller.go:118","msg":"Lost the leader campaign, will unload modules"} {"level":"info","ts":1693818421.2138064,"caller":"controller/controller.go:109","msg":"Start as the leader"}

git-hulk commented 10 months ago

Fixed in unstable branch, please have a try.