A CN of the private network got panic after addValidator

[aidan-kwon]

Issue

• 1600

A CN of a v1.7.3 private network got panic. The network consisted of 3 CN nodes and another CN had been added as a validator. The newly added CN got panic and others generate blocks without the panic issue. After the CN restart, the panic still occurs at the same block number. After the binary change to v1.7.2, the still occurs at the same block number. After rewinding a few blocks with the start-block-num flag, it doesn't show the panic log and synchronization started.

• The private network was using RoundRobin proposerPolicy instead of WeightedRandom.

INFO[02/22,02:40:20 Z] [5] Inserted a new block                      number=25918 hash=d595db…8d6913                                                      txs=0 gas=0 elapsed=1.997ms    processTxs=78ns  finalize=72.308µs  validateState=2.604µs totalWrite=1.84381ms  trieWrite=1.770379ms
panic: sync/atomic: store of nil value into Value

goroutine 2500 [running]:
sync/atomic.(*Value).Store(0xc01003e328, 0x0, 0x0)
    /usr/local/go/src/sync/atomic/value.go:47 +0x12c
github.com/klaytn/klaytn/consensus/istanbul/validator.(*defaultSet).Copy(0xc00fdf3f80, 0x0, 0x0)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/consensus/istanbul/validator/default.go:362 +0x259
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyCommittedSeals(0xc00016e000, 0x1f9ba60, 0xc000203900, 0xc010036900, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/consensus/istanbul/backend/engine.go:307 +0x15c
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyCascadingFields(0xc00016e000, 0x1f9ba60, 0xc000203900, 0xc010036900, 0x0, 0x0, 0x0, 0xc010040000, 0x1)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/consensus/istanbul/backend/engine.go:233 +0x2a6
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyHeader(0xc00016e000, 0x1f9ba60, 0xc000203900, 0xc010036900, 0x0, 0x0, 0x0, 0xc01bd1ec40, 0xc010036900)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/consensus/istanbul/backend/engine.go:199 +0x1b1
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).VerifyHeader(0xc00016e000, 0x1f9ba60, 0xc000203900, 0xc010036900, 0x9ab62eb49a259601, 0xc01b13f5e0, 0xc07d31250f9b0153)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/consensus/istanbul/backend/engine.go:173 +0x5a
github.com/klaytn/klaytn/blockchain.(*BlockChain).insertChain(0xc000203900, 0xc00ea07000, 0x78, 0x78, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/blockchain/blockchain.go:1670 +0x538a
github.com/klaytn/klaytn/blockchain.(*BlockChain).InsertChain(0xc000203900, 0xc00ea07000, 0x78, 0x78, 0x0, 0x0, 0x8)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/blockchain/blockchain.go:1546 +0x67
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).importBlockResults(0xc000149ba0, 0xc00ea06c00, 0x78, 0x78, 0x78, 0x0)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/datasync/downloader/downloader.go:1423 +0x539
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).processFullSyncContent(0xc000149ba0, 0x0, 0x0)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/datasync/downloader/downloader.go:1397 +0x10a
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).spawnSync.func1(0xc000149ba0, 0xc00036fb00, 0xc00f37d700)
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/datasync/downloader/downloader.go:506 +0x5b
created by github.com/klaytn/klaytn/datasync/downloader.(*Downloader).spawnSync
    /go/src/github.com/klaytn/klaytn/kcn-linux-x86_64/rpmbuild/BUILD/kcnd-v1.7.3/datasync/downloader/downloader.go:506 +0x16f
INFO[02/22,02:44:03 Z] [31] Enabling metrics collection              
INFO[02/22,02:44:03 Z] [31] Enabling Prometheus Exporter             
INFO[02/22,02:44:03 Z] [18] Setting connection type                   nodetype=cn conntype=0
INFO[02/22,02:44:03 Z] [18] Customized bootnodes are set             

[yoomee1313]

https://github.com/klaytn/klaytn/pull/1600

[hyunsooda]

@yoomee1313, Didn't you mean to add a reference #1600, not #1608?

[yoomee1313]

@hyunsooda Yes, you're right. Thanks!