[RC Mainnet] crash report

[iAmMichaelConnor]

Background: The node had continuous uptime from main net launch at 00:00 on 17th until a crash at ~17:00 on 19th (CET) (that crash is detailed separately here #8367).

Strangely, my node then spent 20 hours in catchup (I have a status checker that polls every minute and gives me logs to read). It's a 64GB RAM 8 core machine, which was running a block producer and sidecar. It was not running a SNARK worker during the 20 hours leading up to the crash.

The 20 hour catchup period finally ended with a 2nd crash, and it is this 2nd crash for which I attach the report here:

*NOTE:* after this crash, the node restarted and synced within 20 mins. It's been continuously synced ever since. Very strange. Never seen a 20 hour catchup before. Here are the logs immediately before the crash: ``` Mar 20 13:45:07 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:07 UTC [Warn] validation callback timed out before we could respond Mar 20 13:45:07 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:07 UTC [Error] validation timed out :( Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:12 UTC [Info] Updating new available work took 8.548736572265625 ms Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:12 UTC [Error] Error handling committed transaction $cmd: $error Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: cmd: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "data": [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Signed_command", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "payload": { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "common": { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "fee": "0.01", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "fee_token": "1", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "fee_payer_pk": Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "B62qkn8VfJNMwfWooqjWRvFEmMfeajG9XBwejxvQi7H27gTrvgPLB4H", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "nonce": "6", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "valid_until": "4294967295", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "memo": "E4YM2vTHhWEg66xpj52JErHUBU4pZ1yageL4TVDDpTTSsv8mK6YaH" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: }, Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "body": [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Stake_delegation", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Set_delegate", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "delegator": Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "B62qkn8VfJNMwfWooqjWRvFEmMfeajG9XBwejxvQi7H27gTrvgPLB4H", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "new_delegate": Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "B62qp8Vq6n4VHq1LUm9Wd5QKjpKb7umoZ2oU9gpJYuHNUc7t2HGhGUA" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ] Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ] Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: }, Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "signer": "B62qkn8VfJNMwfWooqjWRvFEmMfeajG9XBwejxvQi7H27gTrvgPLB4H", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "signature": Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "7mXN2Qyn4Ya4TQNWpspt3aUg4vSptk8V1yracHyNsVRHpQ7h1ro2cAaNw4uPYayuaVdeejGi5fG1XVUN7YFoQteV4hnfcYpA" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ], Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "status": [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Applied", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "fee_payer_account_creation_fee_paid": null, Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "receiver_account_creation_fee_paid": null, Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "created_token": null Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: }, Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "fee_payer_balance": "1419.97", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "source_balance": "1419.97", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "receiver_balance": "0" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ] Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: error: "Tried to handle a committed transaction in the pool but its nonce doesn't match the head of the queue for that sender" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:12 UTC [Fatal] Unhandled top-level exception: $exn Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: Generating crash report Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: exn: { Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "commit_id": "48401e92d94948c066c03ca76326c065e5cbfc92", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "sexp": [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "monitor.ml.Error", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Failure", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Tried to handle a committed transaction in the pool but its nonce doesn't match the head of the queue for that sender" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ], Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Raised at file \"stdlib.ml\", line 33, characters 17-33", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/lib/network_pool/transaction_pool.ml\", line 521, characters 18-36", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"list.ml\", line 117, characters 24-34", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/list0.ml\" (inlined), line 22, characters 40-81", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/lib/network_pool/transaction_pool.ml\", line 462, characters 8-1023", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/lib/pipe_lib/strict_pipe.ml\", line 159, characters 28-37", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/deferred0.ml\", line 56, characters 64-69", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"src/job_queue.ml\", line 171, characters 6-47", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Caught by monitor coda" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ] Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ], Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "backtrace": [ Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Raised at file \"format.ml\" (inlined), line 242, characters 35-52", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"format.ml\", line 469, characters 8-33", Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: "Called from file \"format.ml\", line 484, characters 6-24" Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: ] Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:12 UTC [Info] Saw block with state hash $state_hash Mar 20 13:45:12 Ubuntu-1804-bionic-64-minimal mina[1419]: state_hash: "3NKyAEH4fdAe3zqnm91farzE5ZvfAkaH9rey9nRVqos9b9a2dy2b" Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:15 UTC [Info] Updating new available work took 7.972478866577148 ms Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:15 UTC [Info] Saw block with state hash $state_hash Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: state_hash: "3NKj7H44kzbTmTU1sWKjZ52EujGjcbRwEBf72aGyBqnRZ28DCBsf" Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:15 UTC [Error] error sending message on stream 19920: $error Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: error: { Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: "commit_id": "48401e92d94948c066c03ca76326c065e5cbfc92", Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: "string": Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: "RPC #35182 failed: \"only wrote 0 out of 203 bytes error: libp2p error error: closed stream\"" Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:15 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:15 UTC [Warn] RPC call error for "get_transition_chain" Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:16 UTC [Error] error sending message on stream 19895: $error Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: error: { Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: "commit_id": "48401e92d94948c066c03ca76326c065e5cbfc92", Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: "string": Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: "RPC #35190 failed: \"only wrote 0 out of 15 bytes error: libp2p error error: closed stream\"" Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: } Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: 2021-03-20 12:45:16 UTC [Warn] RPC call error for "get_transition_chain_proof" Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: ☠ Mina Daemon crashed. Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: The Mina Protocol developers would like to know why! Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: Please: Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: Open an issue: Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: Briefly describe what you were doing and attach the crash report /home/mike/.mina-config/coda_crash_report_2021-03-20_12-45-15.590877.tar.gz [coda_crash_report_2021-03-20_12-45-15.590877.tar.gz](https://github.com/MinaProtocol/mina/files/6178211/coda_crash_report_2021-03-20_12-45-15.590877.tar.gz) Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal mina[1419]: Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal systemd[30300]: mina.service: Main process exited, code=exited, status=1/FAILURE Mar 20 13:45:16 Ubuntu-1804-bionic-64-minimal systemd[30300]: mina.service: Failed with result 'exit-code'. Mar 20 13:45:31 Ubuntu-1804-bionic-64-minimal systemd[30300]: mina.service: Service hold-off time over, scheduling restart. Mar 20 13:45:31 Ubuntu-1804-bionic-64-minimal systemd[30300]: mina.service: Scheduled restart job, restart counter is at 2. Mar 20 13:45:31 Ubuntu-1804-bionic-64-minimal systemd[30300]: Stopped Mina Daemon Service. ```

[iAmMichaelConnor]

coda_crash_report_2021-03-20_12-45-15.590877.tar.gz

[bkase]

This is likely related to a transition frontier desync check out RFC 0028

Since these are rare, we don't plan on implementing this RFC soon.

[bkase]

This could be already fixed, so let's keep our eyes out for this to happen again

[bkase]

This could be already fixed, so let's keep our eyes out for this to happen again

[shimkiv]

Closing this one because of no activity for a long time and I hope that the most recent versions of Mina Daemon behave more stable for you. Please feel free to report new issues should you face any. Thank you.