search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.99k stars 529 forks source link

[Mainnet RC] daemon crashed after failing to produce a block #8303

Open lampardlamps opened 3 years ago

lampardlamps commented 3 years ago

I was running a snark worker on the backup machine and was probably turning snark off too late (~2.5mins before my slot). The journal shows that the node received a few blocks and output 'successfully produced a block', but crashed almost immediately afterwards, and the block was not shown on explorer (not even orphaned). coda_crash_report_2021-03-17_03-54-48.648595.tar.gz

deepthiskumar commented 3 years ago

Failed when adding a locally produced block to the transition frontier. Looks like this is from a duplicate block producer which created an identical block. There is aReceived a block log with state hash 3NKCdQt2pAeLqrPYxLHNs7UeusGGrTL8X6HcaFs8yJH4YAe45oLz 

"timestamp":"2021-03-17 03:54:42.251780Z","level":"Info","source":{"module":"Mina_networking","location":"File \"src/lib/mina_networking/mina_networking.ml\", line 1203, characters 14-29"},"message":"Received a block from $sender","metadata":{"external_transition":{"protocol_state":{"previous_state_hash":"3NLUgbM5PpzExzsvcbxFy5G4PrSNCMNdL7Q8whJiJMdxQ7ZVuUZu","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jwdsK5ukwfWoXNeaeFHPtaF75NqAMsFBSLREWq8DvnkwaqsUsQR","aux_hash":"UmFmaSyqoL7nkcZixhuLN9PfE7PmxW6rYsF6cdJAeqPYZT6Re2","pending_coinbase_aux":"WacefYb9UHzMGR2otu9uPPWE9xA13zeXbLuh93dNtCzsTQdc1T"},"pending_coinbase_hash":"2mzeXxnMGcJ7Qrdd9QetpndiR4QsDVibiFW9FWJ4UtbqNcDYHyuT"},"snarked_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1615953240000"},"consensus_state":{"blockchain_length":"60","epoch_count":"0","min_window_density":"77","sub_window_densities":["2","6","4","5","5","6","6","6","5","4","4"],"last_vrf_output":"E5C9NZiDJ5NIcmbE9ZeYLII5kBMsbJM_p4u78ELjDQA=","total_currency":"805385692840039233","curr_global_slot":{"slot_number":"78","slots_per_epoch":"7140"},"global_slot_since_genesis":"78","staking_epoch_data":{"ledger":{"hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","total_currency":"805385692840039233"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","total_currency":"805385692840039233"},"seed":"2vataxM3VUntahKptPfmG4J7P8ifodhzHyJtRxrPmTcw2L5FrYsx","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NLUgbM5PpzExzsvcbxFy5G4PrSNCMNdL7Q8whJiJMdxQ7ZVuUZu","epoch_length":"61"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmSy1QA6Qbg6Pi9KwudDLFduUGt9dEFkMVuqJwzcherG2fosok6Y","block_creator":"B62qpJavug1VGCBSttepmXr6nh8fvXY5SigbN44ttYDia65vwbTEcq2","coinbase_receiver":"B62qmvHQzJmT2rKE1F9RemenGRG8BfXT1Kurve3eT4iC2HMrWiaVG3H","supercharge_coinbase":false},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"host":"13.56.110.218","peer_id":"12D3KooWBVMWi3NNATYYR95Ttc7k1fKudRpFrZhMHPRuHPZXWm2M","pid":2136,"port":8302,"sender":{"Remote":{"host":"195.201.196.183","peer_id":"12D3KooWHPjzjV3ZgTDuFE6E4wovNBFcWG6w7Wt22qgz96CWJZjG","libp2p_port":8302}},"state_hash":"3NKCdQt2pAeLqrPYxLHNs7UeusGGrTL8X6HcaFs8yJH4YAe45oLz"},"event_id":"b7fa8b10ff17b32b1aa2e2ef508ba811"}

and a Successfully produced a new block log with the same state hash

{"timestamp":"2021-03-17 03:54:44.957881Z","level":"Info","source":{"module":"Block_producer","location":"File \"src/lib/block_producer/block_producer.ml\", line 608, characters 20-35"},"message":"Successfully produced a new block","metadata":{"breadcrumb":{"validated_transition":{"data":{"protocol_state":{"previous_state_hash":"3NLUgbM5PpzExzsvcbxFy5G4PrSNCMNdL7Q8whJiJMdxQ7ZVuUZu","body":{"genesis_state_hash":"3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ","blockchain_state":{"staged_ledger_hash":{"non_snark":{"ledger_hash":"jwdsK5ukwfWoXNeaeFHPtaF75NqAMsFBSLREWq8DvnkwaqsUsQR","aux_hash":"UmFmaSyqoL7nkcZixhuLN9PfE7PmxW6rYsF6cdJAeqPYZT6Re2","pending_coinbase_aux":"WacefYb9UHzMGR2otu9uPPWE9xA13zeXbLuh93dNtCzsTQdc1T"},"pending_coinbase_hash":"2mzeXxnMGcJ7Qrdd9QetpndiR4QsDVibiFW9FWJ4UtbqNcDYHyuT"},"snarked_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","genesis_ledger_hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","snarked_next_available_token":"2","timestamp":"1615953240000"},"consensus_state":{"blockchain_length":"60","epoch_count":"0","min_window_density":"77","sub_window_densities":["2","6","4","5","5","6","6","6","5","4","4"],"last_vrf_output":"E5C9NZiDJ5NIcmbE9ZeYLII5kBMsbJM_p4u78ELjDQA=","total_currency":"805385692840039233","curr_global_slot":{"slot_number":"78","slots_per_epoch":"7140"},"global_slot_since_genesis":"78","staking_epoch_data":{"ledger":{"hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","total_currency":"805385692840039233"},"seed":"2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","epoch_length":"1"},"next_epoch_data":{"ledger":{"hash":"jx7buQVWFLsXTtzRgSxbYcT8EYLS8KCZbLrfDcJxMtyy4thw2Ee","total_currency":"805385692840039233"},"seed":"2vataxM3VUntahKptPfmG4J7P8ifodhzHyJtRxrPmTcw2L5FrYsx","start_checkpoint":"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x","lock_checkpoint":"3NLUgbM5PpzExzsvcbxFy5G4PrSNCMNdL7Q8whJiJMdxQ7ZVuUZu","epoch_length":"61"},"has_ancestor_in_same_checkpoint_window":true,"block_stake_winner":"B62qmSy1QA6Qbg6Pi9KwudDLFduUGt9dEFkMVuqJwzcherG2fosok6Y","block_creator":"B62qpJavug1VGCBSttepmXr6nh8fvXY5SigbN44ttYDia65vwbTEcq2","coinbase_receiver":"B62qmvHQzJmT2rKE1F9RemenGRG8BfXT1Kurve3eT4iC2HMrWiaVG3H","supercharge_coinbase":false},"constants":{"k":"290","slots_per_epoch":"7140","slots_per_sub_window":"7","delta":"0","genesis_state_timestamp":"1615939200000"}}},"protocol_state_proof":"<opaque>","staged_ledger_diff":"<opaque>","delta_transition_chain_proof":"<opaque>","current_protocol_version":"2.0.0","proposed_protocol_version":"<None>"},"hash":"3NKCdQt2pAeLqrPYxLHNs7UeusGGrTL8X6HcaFs8yJH4YAe45oLz"},"staged_ledger":"<opaque>","just_emitted_a_proof":false,"transition_receipt_time":"2021-03-17T03:54:43.074367Z"},"host":"13.56.110.218","peer_id":"12D3KooWBVMWi3NNATYYR95Ttc7k1fKudRpFrZhMHPRuHPZXWm2M","pid":2136,"port":8302},"event_id":"64e2d3e86c37c09b15efdaf7470ce879"}
lampardlamps commented 3 years ago

Note: this is a duplicate block producing machine. Here is the log file: https://1drv.ms/u/s!AlOGZ7BQqAI1iKlj7Z0AN4cSBZiZOg?e=72MzQF Thanks for looking into this!

lampardlamps commented 3 years ago

That same node crashed again, this time without a report. Sorry I removed config before restarting, but could you confirm whether it was because the snark took too much resources from the system and forced the Daemon to be shut down? Mar 17 10:28:36 ip-172-31-13-60 systemd[1181]: Stopping Mina Daemon Service... Mar 17 10:28:36 ip-172-31-13-60 mina[9785]: Using password from environment variable CODA_PRIVKEY_PASS Mar 17 10:28:36 ip-172-31-13-60 mina[4242]: 2021-03-17 10:28:36 UTC [Info] Running async shutdown handler: "Close transition fr Mar 17 10:28:36 ip-172-31-13-60 mina[4242]: 2021-03-17 10:28:36 UTC [Info] Running async shutdown handler: "Remove daemon lockf Mar 17 10:28:36 ip-172-31-13-60 mina[9785]: Daemon stopping Mar 17 10:28:36 ip-172-31-13-60 mina[4242]: 2021-03-17 10:28:36 UTC [Info] Mina process was interrupted by "sigterm" Mar 17 10:28:36 ip-172-31-13-60 mina[4242]: {"timestamp":"2021-03-17 10:28:36.265400Z","level":"Info","source":{"module":"Snark Mar 17 10:28:36 ip-172-31-13-60 mina[4242]: 2021-03-17 10:28:36 UTC [Error] verifier terminated unexpectedly Mar 17 10:28:36 ip-172-31-13-60 systemd[1181]: mina.service: Main process exited, code=exited, status=130/n/a Mar 17 10:28:36 ip-172-31-13-60 systemd[1181]: mina.service: Failed with result 'exit-code'. Mar 17 10:28:36 ip-172-31-13-60 systemd[1181]: Stopped Mina Daemon Service. -- Reboot --

shimkiv commented 2 years ago

@deepthiskumar can you please check if the issue was fixed already?