MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.99k stars 529 forks source link

Fatal: Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference on mainnet #8681

Open garethtdavies opened 3 years ago

garethtdavies commented 3 years ago
{"exn":{"commit_id":"a42bdeef6b0c15ee34616e4df76c882b0c5c7c2a","sexp":["monitor.ml.Error",["Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference(\"ec5df737-2e0c-3d42-b51a-e8df7b88af1a\", \"File \\\"src/lib/transition_router/transition_router.ml\\\", line 546, characters 61-68\")"],["Raised at file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 116, characters 10-57","Called from file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 531, characters 6-26","Called from file \"src/app/archive/archive_lib/diff.ml\", line 75, characters 17-57","Called from file \"src/list.ml\", line 336, characters 13-17","Called from file \"src/list.ml\" (inlined), line 361, characters 15-31","Called from file \"src/app/archive/archive_lib/diff.ml\", line 70, characters 6-420","Called from file \"src/lib/mina_lib/archive_client.ml\", line 73, characters 21-73","Called from file \"src/deferred_list.ml\", line 10, characters 19-26","Called from file \"src/deferred0.ml\", line 49, characters 2-10","Called from file \"src/lib/pipe_lib/broadcast_pipe.ml\", line 91, characters 25-28","Called from file \"src/pipe.ml\", line 827, characters 4-56","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47","Caught by monitor coda"]],"backtrace":["Raised at file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 116, characters 10-57","Called from file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 531, characters 6-26","Called from file \"src/app/archive/archive_lib/diff.ml\", line 75, characters 17-57","Called from file \"src/list.ml\", line 336, characters 13-17","Called from file \"src/list.ml\" (inlined), line 361, characters 15-31","Called from file \"src/app/archive/archive_lib/diff.ml\", line 70, characters 6-420","Called from file \"src/lib/mina_lib/archive_client.ml\", line 73, characters 21-73","Called from file \"src/deferred_list.ml\", line 10, characters 19-26","Called from file \"src/deferred0.ml\", line 49, characters 2-10","Called from file \"src/lib/pipe_lib/broadcast_pipe.ml\", line 91, characters 25-28","Called from file \"src/pipe.ml\", line 827, characters 4-56","Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5","Called from file \"src/job_queue.ml\", line 171, characters 6-47"]},"host":"44.233.155.135","peer_id":"12D3KooWDwMibKmT86Hm6Eub7rbDDWfzv7wCgpoqw8g16qRmKe1r","pid":12,"port":8812}
  |  
psteckler commented 3 years ago

There's no monadic bind between getting the ledger and this failed call, so the mask was detached when obtained. Is there a guarantee of the attachment stability of a ledger in a breadcrumb?

The purpose of the call is to get a receipt chain status, which I believe is not meaningful now. Can we omit this code, somehow?

nholland94 commented 3 years ago

This could be one of those rare race condition bugs we have seen because of the lack of transition frontier synchronization for extensions. This RFC specifies the issue and how to solve it, we just haven't had a chance to take this work yet. https://github.com/MinaProtocol/mina/blob/develop/rfcs/0028-frontier-synchronization.md

deepthiskumar commented 3 years ago

7 nodes on devnet crashed with this error, all around the same time https://console.cloud.google.com/logs/viewer?interval=CUSTOM&project=o1labs-192920&minLogLevel=0&expandAll=false&timestamp=2021-05-18T04:32:59.408000000Z&customFacets=resource.labels.pod_name,jsonPayload.metadata.exn&limitCustomFacetWidth=true&advancedFilter=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22o1labs-192920%22%0Aresource.labels.location%3D%22us-east4%22%0Aresource.labels.cluster_name%3D%22coda-infra-east4%22%0Aresource.labels.namespace_name%3D%22devnet2%22%0Aresource.labels.container_name%3D%22coda%22%20OR%20resource.labels.container_name%3D%22coordinator%22%0A%22unhandled%20%22&scrollTimestamp=2021-05-18T04:17:59.213098668Z&dateRangeStart=2021-05-18T04:00:30.836Z&dateRangeEnd=2021-05-18T04:20:30.860Z

EmreNOP commented 2 years ago

minaprotocol/mina-daemon:1.2.0-fe51f1e-mainnet crash report node crashed after block creation

Generating crash report exn: { "commit_id": "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af", "sexp": [ "monitor.ml.Error", [ "Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference(\"f601d181-8062-3061-bdc0-91e56b866876\", \"File \\"src/lib/mina_lib/mina_lib.ml\\", line 1357, characters 49-56\")" ], [ "Raised at file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 116, characters 10-57", "Called from file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 542, characters 6-26", "Called from file \"src/lib/network_pool/snark_pool.ml\", line 265, characters 18-115", "Called from file \"src/lib/network_pool/snark_pool.ml\", line 429, characters 10-59", "Called from file \"src/lib/network_pool/snark_pool_diff.ml\", line 121, characters 17-65", "Called from file \"src/lib/network_pool/network_pool_base.ml\", line 172, characters 22-68", "Called from file \"src/result.ml\", line 151, characters 9-15", "Caught by monitor coda" ] ], "backtrace": [ "Raised at file \"stdlib.ml\", line 33, characters 17-33", "Called from file \"src/pipe.ml\", line 873, characters 10-13", "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5", "Called from file \"src/job_queue.ml\", line 171, characters 6-47" ] } 2021-11-03 23:15:33 UTC [Fatal] Unhandled top-level exception: $exn Generating crash report exn: { "commit_id": "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af", "sexp": [ "monitor.ml.Error", [ "Failure", "traced task R&snark_pool_verification failed, exception reported to parent monitor" ], [ "Raised at file \"stdlib.ml\", line 33, characters 17-33", "Called from file \"src/pipe.ml\", line 873, characters 10-13", "Called from file \"src/job_queue.ml\" (inlined), line 131, characters 2-5", "Called from file \"src/job_queue.ml\", line 171, characters 6-47", "Caught by monitor coda" ] ], "backtrace": [ "Raised at file \"format.ml\" (inlined), line 242, characters 35-52", "Called from file \"format.ml\", line 469, characters 8-33", "Called from file \"format.ml\", line 484, characters 6-24" ] } 2021-11-03 23:15:34 UTC [Info] Coda daemon is booting up; built with commit "fe51f1ef5cfbcb4d5f559af39080963cf28ff1af" on branch "master"