Deamon crashed when producing block

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

Deamon crashed when producing block:

2023-01-07 10:44:37 UTC [Info] Producing a block in "29" slots
2023-01-07 10:44:37 UTC [Info] Deamon has been running for 10448 mins. Stopping now...
2023-01-07 10:44:38 UTC [Error] Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ("1 slots too late")
    state_hash: "3NKmru8yY1ka27LSjf5t75U7kPTXfVUTK4bHs3QuwXFJeMaAr4RB"
2023-01-07 10:44:38 UTC [Info] Saw block with state hash $state_hash
    state_hash: "3NKmru8yY1ka27LSjf5t75U7kPTXfVUTK4bHs3QuwXFJeMaAr4RB"
2023-01-07 10:44:38 UTC [Info] Running async shutdown handler: "Close transition frontier, if exists"
2023-01-07 10:44:38 UTC [Info] Running async shutdown handler: "Remove daemon lockfile"
2023-01-07 10:45:28 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
    exn: {
  "commit_id": "25388a0fed9695e8e9d04f75f50c2bae1c9c80db",
  "sexp": [
    "monitor.ml.Error",
    [
      "Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference(\"0ccca769-9918-3073-30c4-8976dfb09874\", \"File \\\"src/lib/mina_lib/mina_lib.ml\\\", line 1418, characters 49-56\")"
    ],
    [
      "Raised at Merkle_mask__Masking_merkle_tree.Make.Attached.assert_is_attached in file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 118, characters 10-57",
      "Called from Merkle_mask__Masking_merkle_tree.Make.Attached.location_of_account_batch in file \"src/lib/merkle_mask/masking_merkle_tree.ml\", line 571, characters 6-26",
      "Called from Network_pool__Snark_pool.Make.Resource_pool.T.handle_new_best_tip_ledger.(fun) in file \"src/lib/network_pool/snark_pool.ml\", line 312, characters 29-136",
      "Called from O1trace.trace_recurring in file \"src/lib/o1trace/o1trace.ml\" (inlined), line 39, characters 2-26",
      "Called from Network_pool__Snark_pool.Make.Resource_pool.T.handle_new_best_tip_ledger.(fun) in file \"src/lib/network_pool/snark_pool.ml\", line 308, characters 23-553",
      "Called from Base__Result.map in file \"src/result.ml\", line 116, characters 19-24",
      "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45",
      "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5",
      "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 168, characters 6-47"
    ]
  ],
  "backtrace": [
    "Raised by primitive operation at Core__Core_unix.read_assume_fd_is_nonblocking in file \"src/core_unix.ml\", line 125, characters 2-55",
    "Called from Async_unix__Interruptor.clear.(fun).loop in file \"src/interruptor.ml\", line 74, characters 17-193"
  ]
}
Mina process exited with status code 1

Not sure if the exception caused the issue or was just a result of shutdown.

Steps to Reproduce

No steps to reproduce, seems like a random failure.

Expected Result

Deamon not crashing randomly.

Actual Result

Deamon crashing randomly.

How frequently do you see this issue?

Sometimes

What is the impact of this issue on your ability to run a node?

High

Status

Using password from environment variable MINA_PRIVKEY_PASS
Using password from environment variable MINA_PRIVKEY_PASS
Using password from environment variable MINA_PRIVKEY_PASS
Mina daemon status
-----------------------------------

Global number of accounts:                     138572
Block height:                                  214028
Max observed block height:                     214028
Max observed unvalidated block height:         214028
Local uptime:                                  38m51s
Ledger Merkle root:                            jwprw9tf9oHLpscTiULW6MKA4b6XsGgQDgMNvjVVghjd1Snknab
Protocol state hash:                           3NLjtGqmk2KgAD7mGNEqv5aFNoHVyYcBtbsVro2uo9UW7JZPrzuM
Chain id:                                      5f704cc0c82e0ed70e873f0893d7e06f148524e3f0bdae2afb02e7819a0c24d1
Git SHA-1:                                     25388a0fed9695e8e9d04f75f50c2bae1c9c80db
Configuration directory:                       /root/.mina-config
Peers:                                         35
User_commands sent:                            0
SNARK worker:                                  None
SNARK work fee:                                100000000
Sync status:                                   Synced
Catchup status:
    To build breadcrumb:           0
    To initial validate:           0
    Finished:                      11
    To download:                   0
    Waiting for parent to finish:  0
    To verify:                     0

Block producers running:                       1 (B62qmFf6UZn2sg3j8bYLGmMinzS2FHX6hDM71nFxAfMhvh4hnGBtkBD)
Coinbase receiver:                             Block producer
Best tip consensus time:                       epoch=44, slot=3347
Best tip global slot (across all hard-forks):  317507
Next block will be produced in:                in 43.661m for slot: 317523 slot-since-genesis: 317523 (Generated from consensus at slot: 317493 slot-since-genesis: 317493)
Consensus time now:                            epoch=44, slot=3348
Consensus mechanism:                           proof_of_stake
Consensus configuration:
    Delta:                     0
    k:                         290
    Slots per epoch:           7140
    Slot duration:             3m
    Epoch duration:            14d21h
    Chain start timestamp:     2021-03-17 00:00:00.000000Z
    Acceptable network delay:  3m

Addresses and ports:
    External IP:    78.129.252.11
    Bind IP:        0.0.0.0
    Libp2p PeerID:  12D3KooWFrRjePtbNPUwEQDqeABJpCLqFW5B4v1fBJu46WLaFrxn
    Libp2p port:    8302
    Client port:    8301

Additional information

No response

MinaProtocol / mina

Deamon crashed when producing block #12471