bestChain / daemonStatus.blockchainLength becomes null in graphql & node halt

jun0tpyrc commented 1 week ago

Preliminary Checks

[X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
[X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions

Description

    query {
      daemonStatus {
        blockchainLength
      }
    }

=>

{
  "data": {
    "daemonStatus": {
      "blockchainLength": null
    }
  }
}

Steps to Reproduce

use minaprotocol/mina-daemon:3.0.0-93e0279-buster-mainnet run after sometimes call above graphql

Expected Result

so chain length

Actual Result

blockchainLength: null

Daemon version

minaprotocol/mina-daemon:3.0.0-93e0279-buster-mainnet

How frequently do you see this issue?

Sometimes

What is the impact of this issue on your ability to run a node?

Blocker

Status

(i call this just after node reboot)

Mina daemon status
-----------------------------------

Max observed block height:              364037
Max observed unvalidated block height:  0
Local uptime:                           4m11s
Chain id:                               a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1
Git SHA-1:                              93e02797f72abe2cbf8dfca6f2328e9c8cd76546
Configuration directory:                /root/.mina-config
Peers:                                  38
User_commands sent:                     0
SNARK worker:                           None
SNARK work fee:                         100000000
Sync status:                            Bootstrap
Block producers running:                0
Coinbase receiver:                      Block producer
Consensus time now:                     epoch=1, slot=521
Consensus mechanism:                    proof_of_stake
Consensus configuration:
        Delta:                     0
        k:                         290
        Slots per epoch:           7140
        Slot duration:             3m
        Epoch duration:            14d21h
        Chain start timestamp:     2024-06-05 00:00:00.000000Z
        Acceptable network delay:  3m

Addresses and ports:
        External IP:    <masked>
        Bind IP:        0.0.0.0
        Libp2p PeerID:  12D3KooWGfpwsvVscaXSaNfXSVJai2RfSwAGUpjAsiotJikviLqa
        Libp2p port:    8302
        Client port:    8301

Metrics:
        block_production_delay:             7 (0 0 0 0 0 0 0)
        transaction_pool_diff_received:     11
        transaction_pool_diff_broadcasted:  0
        transactions_added_to_pool:         0
        transaction_pool_size:              0
        snark_pool_diff_received:           1
        snark_pool_diff_broadcasted:        0
        pending_snark_work:                 0
        snark_pool_size:                    0

Additional information

n/a

jun0tpyrc commented 1 week ago

Mina daemon status
-----------------------------------

Max observed block height:              364038
Max observed unvalidated block height:  364038
Local uptime:                           6m19s
Chain id:                               a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1
Git SHA-1:                              93e02797f72abe2cbf8dfca6f2328e9c8cd76546
Configuration directory:                /root/.mina-config
Peers:                                  73
User_commands sent:                     0
SNARK worker:                           None
SNARK work fee:                         100000000
Sync status:                            Bootstrap
Block producers running:                0
Coinbase receiver:                      Block producer
Consensus time now:                     epoch=1, slot=522
Consensus mechanism:                    proof_of_stake
Consensus configuration:
        Delta:                     0
        k:                         290
        Slots per epoch:           7140
        Slot duration:             3m
        Epoch duration:            14d21h
        Chain start timestamp:     2024-06-05 00:00:00.000000Z
        Acceptable network delay:  3m

Addresses and ports:
        External IP:    <masked>
        Bind IP:        0.0.0.0
        Libp2p PeerID:  12D3KooWGfpwsvVscaXSaNfXSVJai2RfSwAGUpjAsiotJikviLqa
        Libp2p port:    8302
        Client port:    8301

Metrics:
        block_production_delay:             7 (0 0 0 0 0 0 0)
        transaction_pool_diff_received:     22
        transaction_pool_diff_broadcasted:  0
        transactions_added_to_pool:         0
        transaction_pool_size:              0
        snark_pool_diff_received:           56
        snark_pool_diff_broadcasted:        0
        pending_snark_work:                 0
        snark_pool_size:                    0

after reboot a while but still same for that call

jun0tpyrc commented 1 week ago

{
  daemonStatus {
    blockchainLength
    chainId
    catchupStatus
  }
}
=>
{
  "data": {
    "daemonStatus": {
      "blockchainLength": null,
      "chainId": "a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1",
      "catchupStatus": null
    }
  }
}

{
  daemonStatus {
    blockchainLength
    chainId
    catchupStatus
  }
  bestChain(maxLength: 10)
}
=> 
{
  "errors": [
    {
      "message": "Could not obtain best chain from transition frontier",
      "path": [
        "bestChain"
      ]
    }
  ],
  "data": {
    "daemonStatus": {
      "blockchainLength": null,
      "chainId": "a7351abc7ddf2ea92d1b38cc8e636c271c1dfd2c081c637f62ebc2af34eb7cc1",
      "catchupStatus": null
    },
    "bestChain": null
  }
}

jun0tpyrc commented 1 week ago

on node

2024-06-20 23:09:21 UTC [Error] Duplicate producer and slot: producer = $block_producer, consensus_time = $consensus_time, previous protocol state hash = $hash, current protocol s[35/335]
h = $current_protocol_state_hash
        block_producer: "B62qocN7WxNyAnWoNTuKZkFCYetthSUPX65Y41hgnkdZUmmSeJ1dZcc"
        consensus_time: { "slot_number": "7663", "slots_per_epoch": "7140" }
        hash: "3NKd1tzh5tXtosK29kc1xsGauYHnFt6Vfj4LQukV7uD7dRjiFbRS"
        current_protocol_state_hash: "3NKg3QzYkzLwXygn8S5z9E8Pnc1LhvUvNefXUYfZSDbth91m1SoT"
2024-06-20 23:09:46 UTC [Warn] RPC call error for "answer_sync_ledger_query"
2024-06-20 23:10:33 UTC [Warn] RPC call error for "answer_sync_ledger_query"
2024-06-20 23:10:40 UTC [Error] error sending message on stream "614": $error
        error: {
  "commit_id": "93e02797f72abe2cbf8dfca6f2328e9c8cd76546",
  "string": "only wrote 0 out of 9 bytes error: libp2p error: stream reset"
}
2024-06-20 23:10:40 UTC [Warn] failed to reset stream (this means it was probably closed successfully): $error
        error: {
  "commit_id": "93e02797f72abe2cbf8dfca6f2328e9c8cd76546",
  "string": "internal RPC error: unknown stream_idx"
}
2024-06-20 23:11:13 UTC [Error] Validation timed out on block $state_hash
        state_hash: "3NLyDs27ZoqKq9mpXwjAXre7khkVGvTbmmN8VbPJ4b3WDy2w7XVY"
2024-06-20 23:11:13 UTC [Warn] validation callback timed out before we could respond
2024-06-20 23:11:13 UTC [Error] libp2p_helper: validation timed out :(
2024-06-20 23:11:13 UTC [Error] Validation timed out on block $state_hash

jun0tpyrc commented 1 week ago

is it possible node shutdown with partial insert of t.table block table above, then hitting this ?!

nholland94 commented 5 days ago

Hello @jun0tpyrc, thanks for submitting an issue.

It is expected behavior for the blockchainLength to be null while your node is still bootstrapping to the chain (which is what your status output indicates). The errors from the logs your posted are unrelated. Some of those errors are benign (such as the duplicate producer message).

I see you've updated the title of this issue to include a node halt but you haven't given information about the node halt. If there is a node halt you wish to report, please open a new issue. For now, I'm closing this issue as the initial bug you are reporting is the expected behavior of the GraphQL API.

MinaProtocol / mina