blockchain同步处理导致死锁

blockchain事件处理限定1000容量
比特元节点断网恢复后大批量同步区块，由于区块处理逻辑互斥，区块处理数超过1000后，事件处理流程堵塞
获取到互斥锁的区块执行协程，在CheckBlock时依赖blockchain GetBlock事件处理（此时事件处理已经堵塞），导致循环依赖死锁

互斥锁1等待协程

937 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x15894cc 0x15894a3 0x15986f4 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#   0x468304    sync.runtime_SemacquireMutex+0x24                   /usr/local/go/src/runtime/sema.go:71
#   0x15894cb   sync.(*RWMutex).RLock+0x4b                      /usr/local/go/src/sync/rwmutex.go:63
#   0x15894a2   github.com/33cn/chain33/blockchain.(*OrphanPool).IsKnownOrphan+0x22 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/orphanpool.go:52
#   0x15986f3   github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0x953 
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:71
#   0x15ab147   github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147  
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
#   0x158fb36   github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6     
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
#   0x159320a   github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca    
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408

互斥锁2等待协程

62 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x4828e5 0x483eb6 0x483e95 0x1598db8 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#   0x468304    sync.runtime_SemacquireMutex+0x24                   /usr/local/go/src/runtime/sema.go:71
#   0x4828e4    sync.(*Mutex).lockSlow+0x164                        /usr/local/go/src/sync/mutex.go:138
#   0x483eb5    sync.(*Mutex).Lock+0x35                         /usr/local/go/src/sync/mutex.go:81
#   0x483e94    sync.(*RWMutex).Lock+0x14                       /usr/local/go/src/sync/rwmutex.go:111
#   0x1598db7   github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x97 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:108
#   0x1598c7b   github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb 
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:103
#   0x15ab147   github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147  github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
#   0x158fb36   github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6     
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
#   0x159320a   github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca    
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408

互斥锁占用协程，等待blockchain事件处理

1 @ 0x43b4b6 0x44b272 0x9c61a5 0x9c6525 0xb6dc02 0xb72965 0xb71019 0x159af53 0x159af20 0x1599bc9 0x15998c5 0x158b2b3 0x1598ff5 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#   0x9c61a4    github.com/33cn/chain33/queue.(*client).WaitTimeout+0x184           
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/queue/client.go:155
#   0x9c6524    github.com/33cn/chain33/queue.(*client).Wait+0x24               
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/queue/client.go:170
#   0xb6dc01    github.com/33cn/chain33/util.CheckBlock+0x81                    
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/exec.go:25
#   0xb72964    github.com/33cn/chain33/util.PreExecBlock+0x14a4                
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/util.go:448
#   0xb71018    github.com/33cn/chain33/util.ExecBlock+0x258                    
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/util.go:273
#   0x159af52   github.com/33cn/chain33/blockchain.execBlock+0x172              
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/exec.go:15
#   0x159af1f   github.com/33cn/chain33/blockchain.(*BlockChain).connectBlock+0x13f     github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:288
#   0x1599bc8   github.com/33cn/chain33/blockchain.(*BlockChain).connectBestChain+0x128     github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:206
#   0x15998c4   github.com/33cn/chain33/blockchain.(*BlockChain).maybeAcceptBlock+0x544     github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:187
#   0x158b2b2   github.com/33cn/chain33/blockchain.(*OrphanPool).ProcessOrphans+0x6b2       github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/orphanpool.go:235
#   0x1598ff4   github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x2d4    github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:123
#   0x1598c7b   github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb     github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:103
#   0x15ab147   github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147      github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
#   0x158fb36   github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6         github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
#   0x159320a   github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca        github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408

33cn / chain33

blockchain同步处理导致死锁 #1305