search

stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 672 forks source link

Miner stopped downloading new blocks after some time #2020

Closed diwakergupta closed 4 years ago

diwakergupta commented 4 years ago

Describe the bug After ~17 hours of running a local miner, my node stopped download / processing new blocks. State of my node:

{
    "peer_version": 385875968,
    "pox_consensus": "5355b7a1c667090a9e7148c7d970160912c99817",
    "burn_block_height": 2867,
    "stable_pox_consensus": "9a6b70b1d146e9bf213769c18c9bd09b2c01daca",
    "stable_burn_block_height": 2866,
    "server_version": "blockstack-core 0.0.1 => 23.0.0.0 (fix/met-miner-relay-with-nats:858f54265, debug build, macos [x86_64])",
    "network_id": 2147483648,
    "parent_network_id": 3669344250,
    "stacks_tip_height": 1659,
    "stacks_tip": "1050a5e8196c1b767ed80e5fb529eedaffb2804fb3bffeb508bbbd0f5e155049",
    "stacks_tip_consensus_hash": "5355b7a1c667090a9e7148c7d970160912c99817",
    "unanchored_tip": "385c10dc2466fb8efac8f4d9c55b5430897b7fea7b16cbe36e0d065098f65174",
    "exit_at_block_height": null
}

Krypton miner at the same time:

{
    "peer_version": 385875968,
    "pox_consensus": "93839e2c40da0be98ff7d03754c39fb076230f9c",
    "burn_block_height": 3254,
    "stable_pox_consensus": "01aa6be4b66e4b28f8e2e940044d88ba1354df85",
    "stable_burn_block_height": 3253,
    "server_version": "blockstack-core 0.0.1 => 23.0.0.0 (master:edd559a+, release build, linux [x86_64])",
    "network_id": 2147483648,
    "parent_network_id": 3669344250,
    "stacks_tip_height": 1900,
    "stacks_tip": "d7bd74b6f2f5bec606d961030d464f30a4b0898be10800e4c6b54c658f23cd5a",
    "stacks_tip_consensus_hash": "93839e2c40da0be98ff7d03754c39fb076230f9c",
    "unanchored_tip": "a37c50f771d040f850f75864fb9ad01abd098bcaff714e6b4d0c30316fc583c4",
    "exit_at_block_height": 10680
}

@friedger also reported a similar issue on his miner.

Expected behavior

Node should continue downloading and processing blocks.

Environment (please complete the following information):

Additional context

As best as I can tell, the P2P thread either crashed or deadlocked at some point. Last messages from that thread (Thread(1)):

INFO [1604029931.209117] [testnet/stacks-node/src/neon_node.rs:1154] [ThreadId(1)] Received burnchain block #2867 including block_commit_op - mw2Pg4jQkLeW1R7WTCHcv
XqnbRVWWcRmnL (87f23eb792393dd188390f1732a69560950cd79825c82c1e6f7dc4277014856f)
DEBG [1604029931.209316] [testnet/stacks-node/src/run_loop/neon.rs:282] [ThreadId(1)] Synchronized burnchain up to block height 2867 (chain tip height is 2867)
DEBG [1604029931.209356] [testnet/stacks-node/src/run_loop/neon.rs:227] [ThreadId(1)] Wait until we reach steady-state before processing more burnchain blocks...
DEBG [1604029931.209388] [testnet/stacks-node/src/syncctl.rs:410] [ThreadId(1)] PoX watchdog in last reward cycle -- sync after 10 seconds
DEBG [1604029941.220596] [src/chainstate/burn/db/sortdb.rs:1852] [ThreadId(1)] Connect/Open  sortdb '/tmp/stacks-testnet-6f86671555c50290/burnchain/db/bitcoin/regt
est/sortition.db/data.db' as 'readwrite', with index as '/tmp/stacks-testnet-6f86671555c50290/burnchain/db/bitcoin/regtest/sortition.db/marf'
DEBG [1604029941.220970] [src/chainstate/stacks/index/storage.rs:770] [ThreadId(1)] Opened TrieFileStorage /tmp/stacks-testnet-6f86671555c50290/burnchain/db/bitcoi
n/regtest/sortition.db/marf;
DEBG [1604029941.232007] [src/burnchains/bitcoin/spv.rs:465] [ThreadId(1)] Initialized block headers at /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.
dat.reorg
DEBG [1604029941.232258] [src/burnchains/bitcoin/spv.rs:634] [ThreadId(1)] Insert 1 headers to /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.dat.reorg
 in range 2851-2852
DEBG [1604029941.232425] [src/burnchains/bitcoin/spv.rs:660] [ThreadId(1)] No header for child block 2852, so will not validate continuity
DEBG [1604029941.232553] [src/burnchains/bitcoin/spv.rs:677] [ThreadId(1)] No header for parent block 2850, so will not validate continuity
DEBG [1604029941.232596] [src/burnchains/bitcoin/spv.rs:551] [ThreadId(1)] Write 1 headers at /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.dat.reorg
at 2852
DEBG [1604029941.234107] [src/burnchains/bitcoin/indexer.rs:595] [ThreadId(1)] Search for reorg'ed Bitcoin headers from 2852 - 2868
DEBG [1604029941.234167] [src/burnchains/bitcoin/indexer.rs:419] [ThreadId(1)] (Re)establish peer connection
DEBG [1604029941.409579] [src/burnchains/bitcoin/network.rs:156] [ThreadId(1)] Begin peer handshake to 35.245.94.44:18444
DEBG [1604029941.409710] [src/burnchains/bitcoin/network.rs:256] [ThreadId(1)] Send version (nonce=2220570373711134917) to 35.245.94.44:18444
DEBG [1604029941.612532] [src/burnchains/bitcoin/network.rs:267] [ThreadId(1)] Handle version -- remote peer blockchain height is 2867
DEBG [1604029941.612611] [src/burnchains/bitcoin/network.rs:285] [ThreadId(1)] Send verack
DEBG [1604029941.612927] [src/burnchains/bitcoin/network.rs:294] [ThreadId(1)] Handle verack
DEBG [1604029941.612973] [src/burnchains/bitcoin/network.rs:167] [ThreadId(1)] Established connection to 35.245.94.44:18444, who has 2867 blocks
DEBG [1604029941.613012] [src/burnchains/bitcoin/spv.rs:817] [ThreadId(1)] Get headers 2852-2867 to /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.dat.
reorg
DEBG [1604029941.613329] [src/burnchains/bitcoin/network.rs:344] [ThreadId(1)] Send GetHeaders 4a5922efd998b7e7587ce7799963d1a468a806c08f4d5372b3557f9637057ac2 for
 2000 headers to 35.245.94.44:18444
DEBG [1604029941.824280] [src/burnchains/bitcoin/network.rs:308] [ThreadId(1)] Handle ping 12552720456432640415
DEBG [1604029941.824329] [src/burnchains/bitcoin/network.rs:311] [ThreadId(1)] Send pong 12552720456432640415
DEBG [1604029942.130729] [src/burnchains/bitcoin/spv.rs:634] [ThreadId(1)] Insert 15 headers to /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.dat.reor
g in range 2852-2867
DEBG [1604029942.131909] [src/burnchains/bitcoin/spv.rs:660] [ThreadId(1)] No header for child block 2867, so will not validate continuity
DEBG [1604029942.132107] [src/burnchains/bitcoin/spv.rs:551] [ThreadId(1)] Write 15 headers at /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.dat.reorg
 at 2853
DEBG [1604029942.134915] [src/burnchains/bitcoin/spv.rs:280] [ThreadId(1)] Validate PoW between blocks 4032-6048
DEBG [1604029942.135112] [src/burnchains/bitcoin/spv.rs:532] [ThreadId(1)] Handled 15 Headers: 61354afe0aeb7dc7fb4e23ae2f0cd0e8441038e399ae361bebaa53b192886eac-4fa
f1658a8fdd126987c93cd120744986f6b7578f76ab2bcac323d3eec80c92a
DEBG [1604029942.135290] [src/burnchains/bitcoin/spv.rs:873] [ThreadId(1)] Request headers for blocks 2867 - 4867 in range 2852 - 2867
DEBG [1604029942.135535] [src/burnchains/bitcoin/network.rs:344] [ThreadId(1)] Send GetHeaders 4faf1658a8fdd126987c93cd120744986f6b7578f76ab2bcac323d3eec80c92a for
 2000 headers to 35.245.94.44:18444
DEBG [1604029942.343181] [src/burnchains/bitcoin/indexer.rs:464] [ThreadId(1)] Message handler indicates to stop
DEBG [1604029942.344527] [src/burnchains/bitcoin/indexer.rs:704] [ThreadId(1)] Found common Bitcoin block ancestor at height 2867: BlockHeader { version: 536870912
, prev_blockhash: d9dd9c719a117a724c2a571f81a2b064dca781cc75b79a19c79eaa01a41fb763, merkle_root: 3b99891462155904d52367cc944b431e9a237233e00d16c43549cdf3a5a73e4d,
time: 1604029851, bits: 545259519, nonce: 2 }
DEBG [1604029942.344636] [src/burnchains/bitcoin/indexer.rs:740] [ThreadId(1)] Bitcoin headers history is consistent up to 2867
DEBG [1604029942.345123] [src/burnchains/burnchain.rs:1052] [ThreadId(1)] Sync headers from 2867
DEBG [1604029942.345169] [src/burnchains/bitcoin/indexer.rs:508] [ThreadId(1)] Sync all headers starting at block 2867
DEBG [1604029942.348159] [src/burnchains/bitcoin/spv.rs:465] [ThreadId(1)] Initialized block headers at /tmp/stacks-testnet-6f86671555c50290/burnchain/spv-headers.
dat
DEBG [1604029942.348211] [src/burnchains/bitcoin/indexer.rs:419] [ThreadId(1)] (Re)establish peer connection
DEBG [1604029942.533508] [src/burnchains/bitcoin/network.rs:156] [ThreadId(1)] Begin peer handshake to 35.245.94.44:18444
DEBG [1604029942.533721] [src/burnchains/bitcoin/network.rs:256] [ThreadId(1)] Send version (nonce=2220570373711134917) to 35.245.94.44:18444
diwakergupta commented 4 years ago

I do have the debug logs (~2-GB uncompressed 😱 ) from this node if helpful. I believe @friedger has logs as well.

jcnelson commented 4 years ago

As best as I can tell, the P2P thread either crashed or deadlocked at some point. Last messages from that thread (Thread(1)):

This isn't the p2p thread -- this is the burnchain downloader thread. Can you share the logs, somehow?

diwakergupta commented 4 years ago

As best as I can tell, the P2P thread either crashed or deadlocked at some point. Last messages from that thread (Thread(1)):

This isn't the p2p thread -- this is the burnchain downloader thread. Can you share the logs, somehow?

Got it, thanks. And sure, here you go @jcnelson : https://www.dropbox.com/s/73agdx1rauwl94t/miner-2020.zip?dl=0

friedger commented 4 years ago

@jcnelson log file (1.5GB) https://xordrive.io/?p=friedger.id_9a234298-ce64-4fad-b427-07475930854f

jcnelson commented 4 years ago

Should be fixed in #2014 (which now fixes #1680)

blockstack-devops commented 5 days ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.