search

btcsuite / btcd

An alternative full node bitcoin implementation written in Go (golang)
https://github.com/btcsuite/btcd/blob/master/README.md
ISC License
6.25k stars 2.37k forks source link

"REORGANIZE: Block is causing a reorganize" causing btcd to crash on testnet #1492

Open kepikoi opened 4 years ago

kepikoi commented 4 years ago

My testnet instance is crashing repeatedly trying to sync block 1,608,354 with following log CHAN: REORGANIZE: Block 00000000d9356478290d6b51b176e0c7b320e4de22090f5c1f0b4b9ff80cbcae is causing a reorganize.. I'm running 0.20.1 with --addrindex enabled.

Here an excerpt from the logs:

019-11-14 10:36:31.975 [INF] RPCS: RPC server listening on 0.0.0.0:18334
2019-11-14 10:36:32.076 [INF] AMGR: Loaded 8694 addresses from file '/datadrive/btcddata/testnet/peers.json'
2019-11-14 10:36:32.092 [INF] CMGR: Server listening on 0.0.0.0:18333
2019-11-14 10:36:32.092 [INF] CMGR: Server listening on [::]:18333
2019-11-14 10:36:32.106 [INF] CMGR: 5 addresses found from DNS seed testnet-seed.bluematt.me
2019-11-14 10:36:32.118 [INF] SYNC: New valid peer 54.37.194.57:18333 (outbound) (/Satoshi:0.17.1/)
2019-11-14 10:36:32.118 [INF] SYNC: Syncing to block height 1608410 from peer 54.37.194.57:18333
2019-11-14 10:36:32.123 [INF] SYNC: New valid peer 159.65.121.59:18333 (outbound) (/Satoshi:0.18.1/)
2019-11-14 10:36:32.125 [INF] CMGR: DNS discovery failed on seed testnet-seed.bitcoin.schildbach.de: lookup testnet-seed.bitcoin.schildbach.de: no such host
2019-11-14 10:36:32.210 [INF] CMGR: 38 addresses found from DNS seed seed.tbtc.petertodd.org
2019-11-14 10:36:32.557 [INF] SYNC: New valid peer 89.40.14.228:18333 (outbound) (/Satoshi:0.18.1/)
2019-11-14 10:36:32.557 [INF] SYNC: New valid peer 165.227.101.132:18333 (outbound) (/Satoshi:0.18.1/)
2019-11-14 10:36:32.557 [INF] SYNC: New valid peer 52.24.233.86:18333 (outbound) (/Satoshi:0.16.2/)
2019-11-14 10:36:32.557 [INF] SYNC: New valid peer 13.229.242.210:18333 (outbound) (/Satoshi:0.13.2/)
2019-11-14 10:36:32.588 [INF] CHAN: REORGANIZE: Block 00000000d9356478290d6b51b176e0c7b320e4de22090f5c1f0b4b9ff80cbcae is causing a reorganize.
2019-11-14 10:37:28.738 [INF] BTCD: Version 0.20.1-beta
2019-11-14 10:37:28.763 [INF] BTCD: Loading block database from '/datadrive/btcddata/testnet/blocks_ffldb'
2019-11-14 10:37:28.857 [INF] BCDB: Detected unclean shutdown - Repairing...
2019-11-14 10:37:28.882 [INF] BCDB: Database sync complete
2019-11-14 10:37:28.882 [INF] BTCD: Block database loaded
2019-11-14 10:37:28.896 [INF] INDX: Transaction index enabled because it is required by the address index
2019-11-14 10:37:28.897 [INF] INDX: Address index is enabled
2019-11-14 10:37:28.897 [INF] INDX: Committed filter index is enabled
2019-11-14 10:37:28.898 [INF] CHAN: Loading block index...
2019-11-14 10:37:52.094 [WRN] CHAN: Unknown new rules activated (bit 1)
2019-11-14 10:37:52.094 [INF] CHAN: Chain state (height 1608353, hash 0000000000259199d1d9bef707e1894e5216a8db6a5a5b593b6fd5e7df7c90a4, totaltx 52906301, work 562889333512071099
2940)
2019-11-14 10:37:52.243 [INF] RPCS: RPC server listening on 0.0.0.0:18334
2019-11-14 10:37:52.318 [INF] AMGR: Loaded 8694 addresses from file '/datadrive/btcddata/testnet/peers.json'
2019-11-14 10:37:52.320 [INF] CMGR: 38 addresses found from DNS seed seed.tbtc.petertodd.org
2019-11-14 10:37:52.339 [INF] CMGR: Server listening on 0.0.0.0:18333
2019-11-14 10:37:52.339 [INF] CMGR: Server listening on [::]:18333
2019-11-14 10:37:52.342 [INF] CMGR: DNS discovery failed on seed testnet-seed.bitcoin.schildbach.de: lookup testnet-seed.bitcoin.schildbach.de: no such host
2019-11-14 10:37:52.506 [INF] CMGR: 5 addresses found from DNS seed testnet-seed.bluematt.me
2019-11-14 10:37:52.663 [INF] SYNC: New valid peer 34.222.102.24:18333 (outbound) (/Satoshi:0.18.0/)
2019-11-14 10:37:52.664 [INF] SYNC: Syncing to block height 1608410 from peer 34.222.102.24:18333
2019-11-14 10:37:52.675 [INF] SYNC: Lost peer 34.222.102.24:18333 (outbound)
2019-11-14 10:37:52.675 [WRN] SYNC: No sync peer candidates available
2019-11-14 10:37:52.865 [INF] SYNC: New valid peer 13.73.0.194:18333 (outbound) (/Satoshi:0.15.0/)
2019-11-14 10:37:52.865 [WRN] SYNC: No sync peer candidates available
2019-11-14 10:37:52.986 [INF] SYNC: New valid peer 35.240.254.96:18333 (outbound) (/Satoshi:0.17.0/)
2019-11-14 10:37:52.986 [INF] SYNC: Syncing to block height 1608410 from peer 35.240.254.96:18333
2019-11-14 10:37:53.041 [INF] SYNC: New valid peer 103.76.36.113:18333 (outbound) (/Satoshi:0.17.0.1(bitcoin-testnet)/)
2019-11-14 10:37:53.737 [INF] CHAN: REORGANIZE: Block 00000000d9356478290d6b51b176e0c7b320e4de22090f5c1f0b4b9ff80cbcae is causing a reorganize.

It seems to be an issue with only one of my identically configured nodes (Ubuntu 18.04.3 LTS)

kepikoi commented 4 years ago

Time heals all wounds the say. Not in this case - still crashing on restart:

atlas@vm-btcd-1:~$ btcd --addrindex --testnet
2019-11-15 10:03:39.795 [INF] BTCD: Version 0.20.1-beta
2019-11-15 10:03:39.807 [INF] BTCD: Loading block database from '/datadrive/btcddata/testnet/blocks_ffldb'
2019-11-15 10:03:39.920 [INF] BCDB: Detected unclean shutdown - Repairing...
2019-11-15 10:03:39.939 [INF] BCDB: Database sync complete
2019-11-15 10:03:39.939 [INF] BTCD: Block database loaded
2019-11-15 10:03:39.953 [INF] INDX: Transaction index enabled because it is required by the address index
2019-11-15 10:03:39.953 [INF] INDX: Address index is enabled
2019-11-15 10:03:39.953 [INF] INDX: Committed filter index is enabled
2019-11-15 10:03:39.954 [INF] CHAN: Loading block index...
2019-11-15 10:04:03.671 [INF] CHAN: Chain state (height 1608353, hash 0000000000259199d1d9bef707e1894e5216a8db6a5a5b593b6fd5e7df7c90a4, totaltx 52906301, work 5628893335120710992940)
2019-11-15 10:04:03.794 [INF] RPCS: RPC server listening on 0.0.0.0:18334
2019-11-15 10:04:03.870 [INF] AMGR: Loaded 8694 addresses from file '/datadrive/btcddata/testnet/peers.json'
2019-11-15 10:04:03.870 [INF] CMGR: Server listening on [::]:18333
2019-11-15 10:04:03.885 [INF] CMGR: 5 addresses found from DNS seed testnet-seed.bluematt.me
2019-11-15 10:04:03.888 [INF] CMGR: DNS discovery failed on seed testnet-seed.bitcoin.schildbach.de: lookup testnet-seed.bitcoin.schildbach.de: no such host
2019-11-15 10:04:03.889 [INF] CMGR: Server listening on 0.0.0.0:18333
2019-11-15 10:04:03.890 [INF] CMGR: 38 addresses found from DNS seed seed.tbtc.petertodd.org
2019-11-15 10:04:03.899 [INF] SYNC: New valid peer 89.39.107.219:18333 (outbound) (/Satoshi:0.17.0/)
2019-11-15 10:04:03.899 [INF] SYNC: Syncing to block height 1608540 from peer 89.39.107.219:18333
2019-11-15 10:04:03.901 [INF] SYNC: New valid peer 80.100.203.151:18333 (outbound) (/Satoshi:0.16.3/)
2019-11-15 10:04:03.918 [INF] SYNC: New valid peer 62.210.66.227:18333 (outbound) (/Satoshi:0.15.0/)
2019-11-15 10:04:03.929 [INF] SYNC: New valid peer 88.99.167.175:18333 (outbound) (/Satoshi:0.18.1/)
2019-11-15 10:04:03.942 [INF] CHAN: REORGANIZE: Block 00000000d9356478290d6b51b176e0c7b320e4de22090f5c1f0b4b9ff80cbcae is causing a reorganize.
fatal error: sync: Unlock of unlocked RWMutex

goroutine 38 [running]:
runtime.throw(0xad7f6b, 0x20)
        /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc014499180 sp=0xc014499150 pc=0x42ef22
sync.throw(0xad7f6b, 0x20)
        /usr/local/go/src/runtime/panic.go:603 +0x35 fp=0xc0144991a0 sp=0xc014499180 pc=0x42eea5
sync.(*RWMutex).Unlock(0xc00007db7c)
        /usr/local/go/src/sync/rwmutex.go:124 +0xa2 fp=0xc0144991d0 sp=0xc0144991a0 pc=0x474cf2
runtime.call32(0x0, 0xc1b760, 0xc01faa87f0, 0x800000008)
        /usr/local/go/src/runtime/asm_amd64.s:519 +0x3b fp=0xc014499200 sp=0xc0144991d0 pc=0x45c0eb
panic(0xa0f7a0, 0x10fa720)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5 fp=0xc014499290 sp=0xc014499200 pc=0x42ea55
runtime.panicindex()
        /usr/local/go/src/runtime/panic.go:44 +0xfb fp=0xc0144992c0 sp=0xc014499290 pc=0x42da0b
github.com/btcsuite/btcd/mempool.(*FeeEstimator).RegisterBlock(0xc0000e6000, 0xc014b01650, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/mempool/estimatefee.go:283 +0xd00 fp=0xc014499648 sp=0xc0144992c0 pc=0x8779d0
github.com/btcsuite/btcd/netsync.(*SyncManager).handleBlockchainNotification(0xc0118240c0, 0xc01d98e500)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1397 +0x1f8 fp=0xc014499718 sp=0xc014499648 pc=0x8ed458
github.com/btcsuite/btcd/netsync.(*SyncManager).handleBlockchainNotification-fm(0xc01d98e500)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1350 +0x34 fp=0xc014499738 sp=0xc014499718 pc=0x8eeaf4
github.com/btcsuite/btcd/blockchain.(*BlockChain).sendNotification(0xc00007db00, 0x1, 0xa76940, 0xc014b01650)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/notifications.go:78 +0xac fp=0xc014499778 sp=0xc014499738 pc=0x7cc58c
github.com/btcsuite/btcd/blockchain.(*BlockChain).connectBlock(0xc00007db00, 0xc00ebecf70, 0xc014b01650, 0xc0149b56b0, 0xc0116a1800, 0x2f, 0x2f, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/chain.go:676 +0x3b8 fp=0xc014499800 sp=0xc014499778 pc=0x7bc368
github.com/btcsuite/btcd/blockchain.(*BlockChain).reorganizeChain(0xc00007db00, 0xc014a91110, 0xc014a910e0, 0xc01369b420, 0x1)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/chain.go:1051 +0x149e fp=0xc014499a30 sp=0xc014499800 pc=0x7be09e
github.com/btcsuite/btcd/blockchain.(*BlockChain).connectBestChain(0xc00007db00, 0xc014b0e070, 0xc014b0e000, 0x0, 0x0, 0x0, 0xc00008bad0)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/chain.go:1205 +0xa69 fp=0xc014499b68 sp=0xc014499a30 pc=0x7bf1e9
github.com/btcsuite/btcd/blockchain.(*BlockChain).maybeAcceptBlock(0xc00007db00, 0xc014b0e000, 0x0, 0x0, 0x0, 0x110f960)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/accept.go:79 +0x22b fp=0xc014499c08 sp=0xc014499b68 pc=0x7b9cbb
github.com/btcsuite/btcd/blockchain.(*BlockChain).ProcessBlock(0xc00007db00, 0xc014b0e000, 0xc000000000, 0xc000010000, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/blockchain/process.go:228 +0x2b2 fp=0xc014499d10 sp=0xc014499c08 pc=0x7ccdf2
github.com/btcsuite/btcd/netsync.(*SyncManager).handleBlockMsg(0xc0118240c0, 0xc01e8619a0)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:685 +0x20c fp=0xc014499ea0 sp=0xc014499d10 pc=0x8e941c
github.com/btcsuite/btcd/netsync.(*SyncManager).blockHandler(0xc0118240c0)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1289 +0x59f fp=0xc014499fd8 sp=0xc014499ea0 pc=0x8ecfdf
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc014499fe0 sp=0xc014499fd8 pc=0x45de01
created by github.com/btcsuite/btcd/netsync.(*SyncManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1514 +0xde

goroutine 1 [chan receive]:
main.btcdMain(0x0, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/btcd.go:170 +0x8f2
main.main()
        /home/atlas/go/src/github.com/btcsuite/btcd/btcd.go:330 +0xd1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 21 [select]:
main.interruptListener.func1(0xc000078120)
        /home/atlas/go/src/github.com/btcsuite/btcd/signal.go:31 +0x107
created by main.interruptListener
        /home/atlas/go/src/github.com/btcsuite/btcd/signal.go:25 +0x58

goroutine 6 [select]:
github.com/btcsuite/goleveldb/leveldb/util.(*BufferPool).drain(0xc0000fe000)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/util/buffer_pool.go:206 +0x121
created by github.com/btcsuite/goleveldb/leveldb/util.NewBufferPool
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/util/buffer_pool.go:237 +0x176

goroutine 23 [select]:
github.com/btcsuite/goleveldb/leveldb.(*DB).compactionError(0xc00007cf00)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db_compaction.go:90 +0xcd
created by github.com/btcsuite/goleveldb/leveldb.openDB
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db.go:141 +0x46b

goroutine 24 [select]:
github.com/btcsuite/goleveldb/leveldb.(*DB).mpoolDrain(0xc00007cf00)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db_state.go:97 +0xe4
created by github.com/btcsuite/goleveldb/leveldb.openDB
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db.go:142 +0x48d

goroutine 25 [select]:
github.com/btcsuite/goleveldb/leveldb.(*DB).tCompaction(0xc00007cf00)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db_compaction.go:804 +0x23a
created by github.com/btcsuite/goleveldb/leveldb.openDB
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db.go:148 +0x5e1

goroutine 26 [select]:
github.com/btcsuite/goleveldb/leveldb.(*DB).mCompaction(0xc00007cf00)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db_compaction.go:751 +0x12e
created by github.com/btcsuite/goleveldb/leveldb.openDB
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db.go:149 +0x603

goroutine 27 [select]:
github.com/btcsuite/goleveldb/leveldb.(*DB).jWriter(0xc00007cf00)
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db_write.go:37 +0xf2
created by github.com/btcsuite/goleveldb/leveldb.openDB
        /home/atlas/go/pkg/mod/github.com/btcsuite/goleveldb@v0.0.0-20160330041536-7834afc9e8cd/leveldb/db.go:150 +0x625

goroutine 9 [select]:
net.cgoLookupIP(0xcd6420, 0xc01faa8080, 0xabeee7, 0x2, 0xadc58b, 0x25, 0x400, 0x20300000000000, 0x80, 0x400, ...)
        /usr/local/go/src/net/cgo_unix.go:217 +0x195
net.(*Resolver).lookupIP(0x110f000, 0xcd6420, 0xc01faa8080, 0xabeee7, 0x2, 0xadc58b, 0x25, 0xc0006f7e90, 0x40d619, 0xc00007dc80, ...)
        /usr/local/go/src/net/lookup_unix.go:96 +0x1a4
net.glob..func1(0xcd6420, 0xc01faa8080, 0xc01369a070, 0xabeee7, 0x2, 0xadc58b, 0x25, 0x8103dc, 0xc00024e3f8, 0x0, ...)
        /usr/local/go/src/net/hook.go:23 +0x72
net.(*Resolver).lookupIPAddr.func1(0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/lookup.go:269 +0x116
internal/singleflight.(*Group).doCall(0x110f010, 0xc01370a0f0, 0xc01e7de000, 0x28, 0xc01faa8140)
        /usr/local/go/src/internal/singleflight/singleflight.go:95 +0x2e
created by internal/singleflight.(*Group).DoChan
        /usr/local/go/src/internal/singleflight/singleflight.go:88 +0x29d

goroutine 10 [syscall]:
net._C2func_getaddrinfo(0xc01e7de060, 0x0, 0xc01003c450, 0xc0006ac030, 0x0, 0x0, 0x0)
        _cgo_gotypes.go:92 +0x55
net.cgoLookupIPCNAME.func1(0xc01e7de060, 0x26, 0x26, 0xc01003c450, 0xc0006ac030, 0x25, 0x42dc9f, 0x7fd500000008)
        /usr/local/go/src/net/cgo_unix.go:154 +0x13e
net.cgoLookupIPCNAME(0xabeee7, 0x2, 0xadc58b, 0x25, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/net/cgo_unix.go:154 +0x176
net.cgoIPLookup(0xc01e7fa060, 0xabeee7, 0x2, 0xadc58b, 0x25)
        /usr/local/go/src/net/cgo_unix.go:206 +0x67
created by net.cgoLookupIP
        /usr/local/go/src/net/cgo_unix.go:216 +0xc7

goroutine 31 [chan receive]:
main.newServer.func6(0xc00007d980)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2907 +0x40
created by main.newServer
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2906 +0x18a9

goroutine 32 [chan receive]:
github.com/btcsuite/btcd/netsync.(*SyncManager).IsCurrent(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1553
main.(*server).handleAddPeerMsg(0xc00007d980, 0xc014081d20, 0xc000114460, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:1650 +0x7e4
main.(*server).peerHandler(0xc00007d980)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2121 +0x7ad
created by main.(*server).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2326 +0x123

goroutine 33 [runnable]:
main.(*server).rebroadcastHandler(0xc00007d980)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2263 +0x175
created by main.(*server).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2338 +0x1a8

goroutine 34 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd580, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000105198, 0x72, 0x0, 0x0, 0xac203a)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000105180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000105180, 0xc01003c0f0, 0x30, 0x30)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0020d6290, 0xc01003c0f0, 0xc000032070, 0xc000032000)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0020d6290, 0xc1b748, 0xc001c72e70, 0xc001c72e68, 0x40dea8)
        /usr/local/go/src/net/tcpsock.go:260 +0x48
crypto/tls.(*listener).Accept(0xc01107ac40, 0xa6f140, 0xc01003c0f0, 0xa06440, 0x10fa5f0)
        /usr/local/go/src/crypto/tls/tls.go:62 +0x37
net/http.(*Server).Serve(0xc0002d7ad0, 0xcd44a0, 0xc01107ac40, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2859 +0x22d
main.(*rpcServer).Start.func3(0xc0002d7ad0, 0xc011e8a840, 0xcd44a0, 0xc01107ac40)
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcserver.go:4112 +0xeb
created by main.(*rpcServer).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcserver.go:4110 +0x291

goroutine 35 [select]:
main.queueHandler(0xc011754e40, 0xc011754ea0, 0xc011754f60)
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcwebsocket.go:158 +0x160
main.(*wsNotificationManager).queueHandler(0xc01effaa40)
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcwebsocket.go:198 +0x41
created by main.(*wsNotificationManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcwebsocket.go:1197 +0x5c

goroutine 36 [select]:
main.(*wsNotificationManager).notificationHandler(0xc01effaa40)
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcwebsocket.go:499 +0x2cf
created by main.(*wsNotificationManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/rpcwebsocket.go:1198 +0x7e

goroutine 37 [select]:
github.com/btcsuite/btcd/addrmgr.(*AddrManager).addressHandler(0xc00003b400)
        /home/atlas/go/src/github.com/btcsuite/btcd/addrmgr/addrmanager.go:346 +0xfe
created by github.com/btcsuite/btcd/addrmgr.(*AddrManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/addrmgr/addrmanager.go:580 +0xf4

goroutine 39 [select]:
net.(*Resolver).lookupIPAddr(0x110f000, 0xcd6460, 0xc00009c010, 0xabeee7, 0x2, 0xadc58b, 0x25, 0x2, 0x0, 0x0, ...)
        /usr/local/go/src/net/lookup.go:275 +0x663
net.(*Resolver).LookupIPAddr(...)
        /usr/local/go/src/net/lookup.go:205
net.LookupIP(0xadc58b, 0x25, 0x6, 0x33e84600, 0x33e8464d00000000, 0xc000047e90, 0x40b8ca)
        /usr/local/go/src/net/lookup.go:191 +0x88
main.btcdLookup(0xadc58b, 0x25, 0xc0000a8a00, 0x7fd531edb008, 0xc0000a8a00, 0x1, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/config.go:1184 +0x167
github.com/btcsuite/btcd/connmgr.SeedFromDNS.func1(0xc1abf8, 0x1103760, 0xc01201c560, 0xadc58b, 0x25)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/seed.go:47 +0x186
created by github.com/btcsuite/btcd/connmgr.SeedFromDNS
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/seed.go:44 +0xd5

goroutine 100 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc0001143c0)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 64 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc000114320)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 104 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc000114460)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 44 [select]:
github.com/btcsuite/btcd/connmgr.(*ConnManager).connHandler(0xc000164510)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:241 +0x1ad
created by github.com/btcsuite/btcd/connmgr.(*ConnManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:518 +0xe3

goroutine 45 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd720, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000104398, 0x72, 0x0, 0x0, 0xac203a)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000104380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000104380, 0x110f960, 0xc000059f60, 0x762fe4)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0000100f8, 0xace010, 0x16, 0xc01201c770)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0000100f8, 0xace010, 0x16, 0xc01201c770, 0x1)
        /usr/local/go/src/net/tcpsock.go:260 +0x48
github.com/btcsuite/btcd/connmgr.(*ConnManager).listenHandler(0xc000164510, 0xcd48a0, 0xc0000100f8)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:494 +0xed
created by github.com/btcsuite/btcd/connmgr.(*ConnManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:525 +0x184

goroutine 46 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd650, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000104418, 0x72, 0x0, 0x0, 0xac203a)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000104400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000104400, 0x110f960, 0xc001c74f60, 0x762fe4)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000010100, 0xace010, 0x16, 0xc01369a060)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000010100, 0xace010, 0x16, 0xc01369a060, 0x1)
        /usr/local/go/src/net/tcpsock.go:260 +0x48
github.com/btcsuite/btcd/connmgr.(*ConnManager).listenHandler(0xc000164510, 0xcd48a0, 0xc000010100)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:494 +0xed
created by github.com/btcsuite/btcd/connmgr.(*ConnManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:525 +0x184

goroutine 75 [select]:
github.com/btcsuite/btcd/peer.(*Peer).stallHandler(0xc01e81e000)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1198 +0x1e2
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2152 +0x27a

goroutine 161 [select]:
github.com/btcsuite/btcd/peer.(*Peer).stallHandler(0xc011ff0f00)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1198 +0x1e2
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2152 +0x27a

goroutine 49 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd3e0, 0x77, 0xc013689c80)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000100298, 0x77, 0xcd6400, 0xc01e7fa2a0, 0xc000100280)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).WaitWrite(...)
        /usr/local/go/src/internal/poll/fd_unix.go:498
net.(*netFD).connect(0xc000100280, 0xcd64a0, 0xc01e7fa2a0, 0x0, 0x0, 0xcca860, 0xc0136cc220, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/net/fd_unix.go:152 +0x284
net.(*netFD).dial(0xc000100280, 0xcd64a0, 0xc01e7fa2a0, 0xcda440, 0x0, 0xcda440, 0xc01003c5d0, 0x0, 0x1, 0xc001c70688)
        /usr/local/go/src/net/sock_posix.go:149 +0xff
net.socket(0xcd64a0, 0xc01e7fa2a0, 0xabf357, 0x3, 0x2, 0x1, 0x0, 0x0, 0xcda440, 0x0, ...)
        /usr/local/go/src/net/sock_posix.go:70 +0x1c0
net.internetSocket(0xcd64a0, 0xc01e7fa2a0, 0xabf357, 0x3, 0xcda440, 0x0, 0xcda440, 0xc01003c5d0, 0x1, 0x0, ...)
        /usr/local/go/src/net/ipsock_posix.go:141 +0x141
net.(*sysDialer).doDialTCP(0xc000100200, 0xcd64a0, 0xc01e7fa2a0, 0x0, 0xc01003c5d0, 0xa0f260, 0x112c670, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:65 +0xc2
net.(*sysDialer).dialTCP(0xc000100200, 0xcd64a0, 0xc01e7fa2a0, 0x0, 0xc01003c5d0, 0xc001c708c8, 0x4f058e, 0xbf6bbbec75056d43)
        /usr/local/go/src/net/tcpsock_posix.go:61 +0xd7
net.(*sysDialer).dialSingle(0xc000100200, 0xcd64a0, 0xc01e7fa2a0, 0xccfb00, 0xc01003c5d0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/dial.go:571 +0x34b
net.(*sysDialer).dialSerial(0xc000100200, 0xcd64a0, 0xc01e7fa2a0, 0xc01369a1b0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/dial.go:539 +0x221
net.(*Dialer).DialContext(0xc00018fb70, 0xcd6460, 0xc00009c010, 0xabf357, 0x3, 0xc0136cc200, 0x12, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/net/dial.go:417 +0x66b
net.(*Dialer).Dial(...)
        /usr/local/go/src/net/dial.go:340
net.DialTimeout(0xabf357, 0x3, 0xc0136cc200, 0x12, 0x6fc23ac00, 0xc001c70c38, 0x40dea8, 0x10, 0x9dd100)
        /usr/local/go/src/net/dial.go:326 +0xb7
main.btcdDial(0xccfb00, 0xc01003c540, 0x1b, 0xc01369a170, 0x1, 0x1)
        /home/atlas/go/src/github.com/btcsuite/btcd/config.go:1169 +0x1b4
github.com/btcsuite/btcd/connmgr.(*ConnManager).Connect(0xc000164510, 0xc01487e730)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:444 +0x11b
github.com/btcsuite/btcd/connmgr.(*ConnManager).NewConnReq(0xc000164510)
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:402 +0x2b3
created by github.com/btcsuite/btcd/connmgr.(*ConnManager).Start
        /home/atlas/go/src/github.com/btcsuite/btcd/connmgr/connmanager.go:530 +0x1c6

goroutine 181 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd240, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000100498, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000100480, 0xc01e03f0c0, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000100480, 0xc01e03f0c0, 0x18, 0x18, 0x0, 0x18, 0xc01e03f0c0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc002189208, 0xc01e03f0c0, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:177 +0x69
io.ReadAtLeast(0xcca3c0, 0xc002189208, 0xc01e03f0c0, 0x18, 0x18, 0x18, 0xc000100480, 0xc01ed65dd0, 0x71)
        /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
github.com/btcsuite/btcd/wire.readMessageHeader(0xcca3c0, 0xc002189208, 0x1c18428400000000, 0x73726576b2db3bd9, 0x6e6f69, 0xc0145f8cc8)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:204 +0x88
github.com/btcsuite/btcd/wire.ReadMessageWithEncodingN(0xcca3c0, 0xc002189208, 0x709110b0001117d, 0x2, 0xc002189208, 0xc01e7f4300, 0x40a5fc, 0xc000012000, 0xa184e0, 0xa99a00, ...)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:344 +0x50
github.com/btcsuite/btcd/peer.(*Peer).readMessage(0xc01e083500, 0xc000000002, 0x0, 0x0, 0x89, 0x0, 0x0, 0x1117d00000000, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1009 +0xb3
github.com/btcsuite/btcd/peer.(*Peer).readRemoteVersionMsg(0xc01e083500, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1877 +0x4a
github.com/btcsuite/btcd/peer.(*Peer).negotiateOutboundProtocol(0xc01e083500, 0x0, 0x21a40)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2113 +0x50
github.com/btcsuite/btcd/peer.(*Peer).start.func1(0xc01e083500, 0xc01e0b4240)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2133 +0x77
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2129 +0xf8

goroutine 90 [select]:
github.com/btcsuite/btcd/peer.(*Peer).stallHandler(0xc011ff0600)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1198 +0x1e2
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2152 +0x27a

goroutine 85 [select]:
github.com/btcsuite/btcd/peer.(*Peer).stallHandler(0xc011ff0900)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1198 +0x1e2
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2152 +0x27a

goroutine 131 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd4b0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000100198, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000100180, 0xc01360b500, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000100180, 0xc01360b500, 0x18, 0x18, 0x0, 0x18, 0xc01360b500)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00208db90, 0xc01360b500, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:177 +0x69
io.ReadAtLeast(0xcca3c0, 0xc00208db90, 0xc01360b500, 0x18, 0x18, 0x18, 0xc000100180, 0xc012f72bd0, 0x71)
        /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
github.com/btcsuite/btcd/wire.readMessageHeader(0xcca3c0, 0xc00208db90, 0x1c18428400000000, 0x73726576ca1e84f2, 0x6e6f69, 0xc014123cc8)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:204 +0x88
github.com/btcsuite/btcd/wire.ReadMessageWithEncodingN(0xcca3c0, 0xc00208db90, 0x709110b0001117d, 0x2, 0xc00208db90, 0xc01353dc80, 0x40a5fc, 0xc000012000, 0xa184e0, 0xa99a00, ...)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:344 +0x50
github.com/btcsuite/btcd/peer.(*Peer).readMessage(0xc01355f800, 0xc000000002, 0x0, 0x0, 0x89, 0x0, 0x0, 0x1117d00000000, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1009 +0xb3
github.com/btcsuite/btcd/peer.(*Peer).readRemoteVersionMsg(0xc01355f800, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1877 +0x4a
github.com/btcsuite/btcd/peer.(*Peer).negotiateOutboundProtocol(0xc01355f800, 0xc1b4b8, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2113 +0x50
github.com/btcsuite/btcd/peer.(*Peer).start.func1(0xc01355f800, 0xc0135d67e0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2133 +0x77
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2129 +0xf8

goroutine 80 [select]:
github.com/btcsuite/btcd/peer.(*Peer).stallHandler(0xc01e81e300)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1198 +0x1e2
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2152 +0x27a

goroutine 113 [select]:
github.com/btcsuite/btcd/peer.(*Peer).start(0xc01355f800, 0x1, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2138 +0x192
github.com/btcsuite/btcd/peer.(*Peer).AssociateConnection.func1(0xc01355f800)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2188 +0x2f
created by github.com/btcsuite/btcd/peer.(*Peer).AssociateConnection
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2187 +0xbc

goroutine 12 [select]:
net.(*netFD).connect.func2(0xcd64a0, 0xc01e7fa2a0, 0xc000100280, 0xc01416e360, 0xc01416e300)
        /usr/local/go/src/net/fd_unix.go:129 +0xba
created by net.(*netFD).connect
        /usr/local/go/src/net/fd_unix.go:128 +0x25a

goroutine 179 [select]:
github.com/btcsuite/btcd/peer.(*Peer).start(0xc01e083500, 0x1, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2138 +0x192
github.com/btcsuite/btcd/peer.(*Peer).AssociateConnection.func1(0xc01e083500)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2188 +0x2f
created by github.com/btcsuite/btcd/peer.(*Peer).AssociateConnection
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2187 +0xbc

goroutine 91 [chan receive]:
github.com/btcsuite/btcd/netsync.(*SyncManager).IsCurrent(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1553
main.(*serverPeer).OnGetHeaders(0xc000114320, 0xc011ff0600, 0xc01eecf240)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:750 +0xb5
github.com/btcsuite/btcd/peer.(*Peer).inHandler(0xc011ff0600)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1456 +0x14dd
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2153 +0x29f

goroutine 86 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edd0a0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000100698, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000100680, 0xc01e9e83e0, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000100680, 0xc01e9e83e0, 0x18, 0x18, 0x0, 0x18, 0xc01e9e83e0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0020d6438, 0xc01e9e83e0, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:177 +0x69
io.ReadAtLeast(0xcca3c0, 0xc0020d6438, 0xc01e9e83e0, 0x18, 0x18, 0x18, 0xc014ab0120, 0x0, 0xc0140fcc58)
        /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
github.com/btcsuite/btcd/wire.readMessageHeader(0xcca3c0, 0xc0020d6438, 0xc014ab0120, 0x20001117d, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:204 +0x88
github.com/btcsuite/btcd/wire.ReadMessageWithEncodingN(0xcca3c0, 0xc0020d6438, 0x709110b0001117d, 0x2, 0xc0020d6438, 0xc0140fce68, 0x44cccc, 0x1113e20, 0xc01370aaa8, 0xc0136e0101, ...)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:344 +0x50
github.com/btcsuite/btcd/peer.(*Peer).readMessage(0xc011ff0900, 0x4500000002, 0x110f900, 0xc01e9c0b40, 0xc01e9c0b48, 0x8, 0x8, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1009 +0xb3
github.com/btcsuite/btcd/peer.(*Peer).inHandler(0xc011ff0900)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1334 +0xb9
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2153 +0x29f

goroutine 81 [IO wait]:
internal/poll.runtime_pollWait(0x7fd531edcfd0, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000100798, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000100780, 0xc01e9e8060, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000100780, 0xc01e9e8060, 0x18, 0x18, 0x0, 0x18, 0xc01e9e8060)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0006ac048, 0xc01e9e8060, 0x18, 0x18, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:177 +0x69
io.ReadAtLeast(0xcca3c0, 0xc0006ac048, 0xc01e9e8060, 0x18, 0x18, 0x18, 0xc014ac3b90, 0x0, 0xc0140f7c58)
        /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
        /usr/local/go/src/io/io.go:329
github.com/btcsuite/btcd/wire.readMessageHeader(0xcca3c0, 0xc0006ac048, 0xc014ac3b90, 0x20001117d, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:204 +0x88
github.com/btcsuite/btcd/wire.ReadMessageWithEncodingN(0xcca3c0, 0xc0006ac048, 0x709110b0001117d, 0x2, 0xc0006ac048, 0xc0140f7e68, 0x44cccc, 0x1113e20, 0xc01487eaf8, 0xc0136e0101, ...)
        /home/atlas/go/src/github.com/btcsuite/btcd/wire/message.go:344 +0x50
github.com/btcsuite/btcd/peer.(*Peer).readMessage(0xc01e81e300, 0x4500000002, 0x110f900, 0xc01e9c08c8, 0xc01e9c08f0, 0x8, 0x8, 0x0, 0x0)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1009 +0xb3
github.com/btcsuite/btcd/peer.(*Peer).inHandler(0xc01e81e300)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1334 +0xb9
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2153 +0x29f

goroutine 69 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc0000aa820)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 73 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc0000aa960)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 76 [chan receive]:
main.(*serverPeer).OnBlock(0xc0000aa820, 0xc01e81e000, 0xc000100980, 0xc0000d9500, 0x538d, 0x538d)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:581 +0x139
github.com/btcsuite/btcd/peer.(*Peer).inHandler(0xc01e81e000)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1426 +0xd9c
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2153 +0x29f

goroutine 77 [select]:
github.com/btcsuite/btcd/peer.(*Peer).queueHandler(0xc01e81e000)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1574 +0x21f
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2154 +0x2c4

goroutine 78 [select]:
github.com/btcsuite/btcd/peer.(*Peer).outHandler(0xc01e81e000)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1708 +0xea
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2155 +0x2e9

goroutine 79 [select]:
github.com/btcsuite/btcd/peer.(*Peer).pingHandler(0xc01e81e000)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1782 +0xef
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2156 +0x30e

goroutine 82 [select]:
github.com/btcsuite/btcd/peer.(*Peer).queueHandler(0xc01e81e300)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1574 +0x21f
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2154 +0x2c4

goroutine 83 [select]:
github.com/btcsuite/btcd/peer.(*Peer).outHandler(0xc01e81e300)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1708 +0xea
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2155 +0x2e9

goroutine 84 [select]:
github.com/btcsuite/btcd/peer.(*Peer).pingHandler(0xc01e81e300)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1782 +0xef
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2156 +0x30e

goroutine 87 [select]:
github.com/btcsuite/btcd/peer.(*Peer).queueHandler(0xc011ff0900)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1574 +0x21f
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2154 +0x2c4

goroutine 88 [select]:
github.com/btcsuite/btcd/peer.(*Peer).outHandler(0xc011ff0900)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1708 +0xea
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2155 +0x2e9

goroutine 89 [select]:
github.com/btcsuite/btcd/peer.(*Peer).pingHandler(0xc011ff0900)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1782 +0xef
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2156 +0x30e

goroutine 92 [select]:
github.com/btcsuite/btcd/peer.(*Peer).queueHandler(0xc011ff0600)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1574 +0x21f
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2154 +0x2c4

goroutine 93 [select]:
github.com/btcsuite/btcd/peer.(*Peer).outHandler(0xc011ff0600)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1708 +0xea
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2155 +0x2e9

goroutine 94 [select]:
github.com/btcsuite/btcd/peer.(*Peer).pingHandler(0xc011ff0600)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1782 +0xef
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2156 +0x30e

goroutine 130 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc0134ec3c0)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 180 [chan receive]:
github.com/btcsuite/btcd/peer.(*Peer).WaitForDisconnect(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2200
main.(*server).peerDoneHandler(0xc00007d980, 0xc0133cdf40)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2062 +0x55
created by main.(*server).outboundPeerConnected
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:2056 +0x3f5

goroutine 162 [chan receive]:
github.com/btcsuite/btcd/netsync.(*SyncManager).IsCurrent(...)
        /home/atlas/go/src/github.com/btcsuite/btcd/netsync/manager.go:1553
main.(*serverPeer).OnGetHeaders(0xc000114460, 0xc011ff0f00, 0xc017a97540)
        /home/atlas/go/src/github.com/btcsuite/btcd/server.go:750 +0xb5
github.com/btcsuite/btcd/peer.(*Peer).inHandler(0xc011ff0f00)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1456 +0x14dd
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2153 +0x29f

goroutine 163 [select]:
github.com/btcsuite/btcd/peer.(*Peer).queueHandler(0xc011ff0f00)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1574 +0x21f
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2154 +0x2c4

goroutine 164 [select]:
github.com/btcsuite/btcd/peer.(*Peer).outHandler(0xc011ff0f00)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1708 +0xea
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2155 +0x2e9

goroutine 165 [select]:
github.com/btcsuite/btcd/peer.(*Peer).pingHandler(0xc011ff0f00)
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:1782 +0xef
created by github.com/btcsuite/btcd/peer.(*Peer).start
        /home/atlas/go/src/github.com/btcsuite/btcd/peer/peer.go:2156 +0x30e
atlas@vm-btcd-1:~$

What could this mean fatal error: sync: Unlock of unlocked RWMutex?

zquestz commented 4 years ago

I am also seeing a similar issue. It does not happen on every reorg, but I have seen it crash my testnet node twice. The stack trace is nearly identical but I can't seem to pinpoint why it is occurring.

zquestz commented 4 years ago

Not sure if this will help, but since both of our stack traces include 2 notifications, I think this might do the trick if you want to port it over to btcd.

https://github.com/gcash/bchd/pull/308

cubancorona commented 4 years ago

Thank you for reporting this significant issue!

I suspect that line 283 (cf. line 278) in btcd\mempool\estimatefee.go is actually the culprit, potentially due to a negative index. As you can see, the panic occurs before the locking issue.

zquestz commented 4 years ago

@cubancorona can you please let me know if you have a fix for the mempool code? Is this something you are actively working on?

Also, I would assume it isn't just a negative index. Usually go will output that error explicitly!

~/Downloads » go run test.go
panic: runtime error: index out of range [-1]

goroutine 1 [running]:
main.main()
    /Users/joshellithorpe/Downloads/test.go:8 +0x1f
exit status 2
kepikoi commented 4 years ago

Another of my nodes crashed with the same error

2019-11-20 08:23:33.079 [INF] CHAN: Adding orphan block 000000000000021a9e3f0e0c4069dca47390ed5cdd5ec2826383345660803fed with parent 00000000001e473bcef526a18b4305397645f603cddd01743aa6589b318a80e7 2019-11-20 08:23:33.156 [INF] CHAN: FORK: Block 00000000001e473bcef526a18b4305397645f603cddd01743aa6589b318a80e7 forks the chain at height 1609136/block 0000000000000021e0eb7d0971598a50d8ddf9b4be58d242aaadd6802f0a7288, but does not cause a reorganize 2019-11-20 08:23:33.157 [INF] CHAN: REORGANIZE: Block 000000000000021a9e3f0e0c4069dca47390ed5cdd5ec2826383345660803fed is causing a reorganize.

In this case I was able to restart btcd and continue syncing but this is really getting unsettling here...

Would you suggest a downgrade to mitigate this?

cubancorona commented 4 years ago

Hi, all, I have been testing a potential remedy for the last few days:

https://github.com/cubancorona/btcd/commit/804b9a6bbfcd6422905cd180c0c434d8e2d87307

kepikoi commented 4 years ago

Can we please have an update on this? Another two testnet3 nodes died with CHAN: REORGANIZE: Block 00000000000002015b7828053c6021c4d5ba6aa176b0d21aebe703975773aafc is causi ng a reorganize. This is getting a major pain to operate testnet nodes via btcd

cubancorona commented 4 years ago

Hi, @kepikoi, thank you kindly for requesting an update on this potentially important issue!

What commit or btcd version are you running?

If you can duplicate the issue, would you mind helping me test the following commit, containing major modifications, some of which are intentionally speculative? https://github.com/cubancorona/btcd/commit/5b03458758a8ff11f6e953eb9c39f4d5db797ba3

Thank you again for following up!

cubancorona commented 4 years ago

@zquestz, please forgive me for not responding directly to your earlier request.

I agree that, if my analysis is correct, the stack trace was a red herring in some significant mannerisms.

I have not been successfully duplicating the issue, so it's hard to speculate! I am hoping some additional testing will help me confirm the issue. Are you able and willing to help me further isolate the potential issue? Itadakimasu!

zquestz commented 4 years ago

@cubancorona I have not experienced this issue since I put https://github.com/gcash/bchd/pull/308 into bchd. Since it is hard to reproduce I am not 100% sure it fixes the issue, but I haven't seen it since I put in the fix. =)

kepikoi commented 4 years ago

@cubancorona currently I'm running https://github.com/btcsuite/btcd/commit/f3ec13030e4e828869954472cbc51ac36bee5c1d. I did restart btcd and unlike in https://github.com/btcsuite/btcd/issues/1492#issue-522780952 this time both nodes were able to process 00000000000002015b7828053c6021c4d5ba6aa176b0d21aebe703975773aafc without crashing.

I'm willing to try https://github.com/cubancorona/btcd/commit/5b03458758a8ff11f6e953eb9c39f4d5db797ba3 on a designated instance and will report the next time a REORGANIZE related crash occurs.

kepikoi commented 4 years ago

@cubancorona In the past weeks I had at least one occurrence on my vms but the https://github.com/cubancorona/btcd/commit/5b03458758a8ff11f6e953eb9c39f4d5db797ba3 instance ran stable. So maybe it fixed it?

kepikoi commented 3 years ago

A few hours ago all (2) of my 0.21.0-beta @https://github.com/btcsuite/btcd/commit/f3ec13030e4e828869954472cbc51ac36bee5c1d main net nodes crashed due to a REORGANIZE event. Had to restart asap so all I can provide is a screenshot image

jonathanbier commented 3 years ago

@kepikoi

https://forkmonitor.info/ is having the same issue

Sjors commented 3 years ago

This happened on the @bitmexresearch ForkMonitor node too. I think it happened before too.

Running version 0.21.0-beta.

...
2021-03-20 03:15:21.856 [INF] SYNC: Processed 1 block in the last 3m8.57s (466 transactions, height 675406, 2021-03-20 03:14:29 +0000 UTC)
2021-03-20 03:17:29.015 [INF] SYNC: Processed 1 block in the last 2m7.15s (251 transactions, height 675407, 2021-03-20 03:16:08 +0000 UTC)
2021-03-20 03:19:47.867 [INF] CHAN: Adding orphan block 0000000000000000000a8be08efb1c929358d3fa323594f77a8d51c385a09302 with parent 0000000000000000000a5bf842478fb470e340a2db4e882f6861696e7e00036e
2021-03-20 03:19:48.566 [INF] CHAN: FORK: Block 0000000000000000000a5bf842478fb470e340a2db4e882f6861696e7e00036e forks the chain at height 675406/block 0000000000000000000af3ee8710894ba53c8d2517c7c191ff78610bf08d7e28, but does not cause a reorganize
2021-03-20 03:19:48.615 [INF] CHAN: REORGANIZE: Block 0000000000000000000a8be08efb1c929358d3fa323594f77a8d51c385a09302 is causing a reorganize.