search

MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.99k stars 525 forks source link

Mina crashing with `Unhandled top-level exception` - Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference #12437

Open kucharskim opened 1 year ago

kucharskim commented 1 year ago

Preliminary Checks

Description

from logs:

2022-12-24 03:11:45 UTC [Info] Deamon has been running for 10414 mins. Stopping now...
2022-12-24 03:11:52 UTC [Info] Running async shutdown handler: "Close transition frontier, if exists"
2022-12-24 03:11:52 UTC [Info] Running async shutdown handler: "Remove daemon lockfile"
2022-12-24 03:12:38 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
        exn: {
  "commit_id": "25388a0fed9695e8e9d04f75f50c2bae1c9c80db",
  "sexp": [
    "monitor.ml.Error",
    [
      "Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference(\"0f4a1cb4-62fe-32ba-75cf-831ded748b50\", \"File \\\"src/lib/mina_lib/mina_lib.ml\\\", line 1418, characters 49-56\")"
    ],
    [
      "Raised at Merkle_mask__Masking_merkle_tree.Make.Attached.assert_is_attached in file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 118, characters 10-57",
      "Called from Merkle_mask__Masking_merkle_tree.Make.Attached.depth in file \"src/lib/merkle_mask/masking_merkle_tree.ml\" (inlined), line 133, characters 18-38",
      "Called from Staged_ledger.T.apply_diff in file \"src/lib/staged_ledger/staged_ledger.ml\", line 846, characters 45-68",
      "Called from Staged_ledger.T.apply.(fun) in file \"src/lib/staged_ledger/staged_ledger.ml\", line 1065, characters 6-282",
      "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69",
      "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5",
      "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 168, characters 6-47",
      "Caught by monitor coda"
    ]
  ],
  "backtrace": [
    "Raised at Yojson.Safe.__ocaml_lex_read_object_sep_rec in file \"lib/read.mll\", line 712, characters 13-36",
    "Called from Yojson.Safe.__ocaml_lex_read_json_rec in file \"lib/read.mll\", line 241, characters 23-47"
  ]
}
2022-12-24 03:12:38 UTC [Error] Validation error: external transition with state hash $state_hash was rejected for reason "verifier error"
        state_hash: "3NK3V98THfBKPBH1BUByFH5tHjX1NWTy1ADyKK7eXBx4v7zZRpsV"
2022-12-24 03:12:38 UTC [Error] Error in verifier verifying blockchain proof for $state_hash: $error
        state_hash: "3NK3V98THfBKPBH1BUByFH5tHjX1NWTy1ADyKK7eXBx4v7zZRpsV"
        error: {
  "commit_id": "25388a0fed9695e8e9d04f75f50c2bae1c9c80db",
  "string": "verify_blockchain_snarks timeout"
}
Mina process exited with status code 1

Steps to Reproduce

No repro, crashed randomly, triggered alarms.

Expected Result

Should not crash with Unhandled top-level exception

Actual Result

Crashed

How frequently do you see this issue?

Sometimes

What is the impact of this issue on your ability to run a node?

Medium

Status

This is half an hour later:

root@mina02:~# docker exec -ti mina_mina_1 mina client status
Using password from environment variable MINA_PRIVKEY_PASS
Using password from environment variable MINA_PRIVKEY_PASS
Using password from environment variable MINA_PRIVKEY_PASS
Mina daemon status
-----------------------------------

Global number of accounts:                     136893
Block height:                                  209731
Max observed block height:                     209731
Max observed unvalidated block height:         209731
Local uptime:                                  36m24s
Ledger Merkle root:                            jwpFcghZCGhDcptMP3TcsoqZffV1kQynFhdHjBezVxWwcTbCUdp
Protocol state hash:                           3NLa3Nw4XugysgEtbbuNwXgUHrk6n9mRkpauXpoH4VgeHUCsR6Bh
Chain id:                                      5f704cc0c82e0ed70e873f0893d7e06f148524e3f0bdae2afb02e7819a0c24d1
Git SHA-1:                                     25388a0fed9695e8e9d04f75f50c2bae1c9c80db
Configuration directory:                       /root/.mina-config
Peers:                                         63
User_commands sent:                            0
SNARK worker:                                  None
SNARK work fee:                                100000000
Sync status:                                   Synced
Catchup status:                                
        To build breadcrumb:           0
        To initial validate:           0
        Finished:                      13
        To download:                   0
        Waiting for parent to finish:  0
        To verify:                     0

Block producers running:                       1 (B62qmFf6UZn2sg3j8bYLGmMinzS2FHX6hDM71nFxAfMhvh4hnGBtkBD)
Coinbase receiver:                             Block producer
Best tip consensus time:                       epoch=43, slot=3613
Best tip global slot (across all hard-forks):  310633
Next block will be produced in:                in 25.664m for slot: 310645 slot-since-genesis: 310645 (Generated from consensus at slot: 310622 slot-since-genesis: 310622)
Consensus time now:                            epoch=43, slot=3616
Consensus mechanism:                           proof_of_stake
Consensus configuration:                       
        Delta:                     0
        k:                         290
        Slots per epoch:           7140
        Slot duration:             3m
        Epoch duration:            14d21h
        Chain start timestamp:     2021-03-17 00:00:00.000000Z
        Acceptable network delay:  3m

Addresses and ports:                           
        External IP:    78.129.252.11
        Bind IP:        0.0.0.0
        Libp2p PeerID:  12D3KooWRr51cq6hHX4RP1cA8ts4KKvWSujcqGVhRJ2HUaRnNDmJ
        Libp2p port:    8302
        Client port:    8301



### Additional information

No additional information at the moment.
kucharskim commented 1 year ago

Version: minaprotocol/mina-daemon:1.3.1.2-25388a0-bullseye-mainnet

shimkiv commented 1 year ago

@deepthiskumar can you please take a look.