INTERNAL_ERROR, StorageError(MissingTrieValue(TrieStorage, BgF6M7oAXo4DbjP3wGfmxNzRfbAhRqSz8R7hnRam7eHm))

[qnduncan]

Describe the bug

Response with "INTERNAL_ERROR" from an archive node.

{
  "jsonrpc": "2.0",
  "error": {
    "name": "INTERNAL_ERROR",
    "cause": {
      "info": {
        "error_message": "StorageError(MissingTrieValue(TrieStorage, BgF6M7oAXo4DbjP3wGfmxNzRfbAhRqSz8R7hnRam7eHm))"
      },
      "name": "INTERNAL_ERROR"
    },
    "code": -32000,
    "message": "Server error",
    "data": "The node reached its limits. Try again later. More details: StorageError(MissingTrieValue(TrieStorage, BgF6M7oAXo4DbjP3wGfmxNzRfbAhRqSz8R7hnRam7eHm))"
  },
  "id": "dontcare"
}

Request:

{
  "jsonrpc": "2.0",
  "id": "dontcare",
  "method": "query",
  "params": {
    "request_type": "call_function",
    "finality": "final",
    "account_id": "bisontrails.poolv1.near",
    "block_id": 114580308,
    "method_name": "get_reward_fee_fraction",
    "args_base64": ""
  }
}

To Reproduce

1. Download latest snapshot for split-storage archive node (cold-data and hot-data) as described in https://near-nodes.io/intro/node-data-snapshots.
2. After snapshots are downloaded, start the neard and send the request above to neard.

Expected behavior We expect to get the response shown below which is returned by archival-rpc.mainnet.near.org.

{"jsonrpc":"2.0","result":{"result":[123,34,110,117,109,101,114,97,116,111,114,34,58,49,48,44,34,100,101,110,111,109,105,110,97,116,111,114,34,58,49,48,48,125],"logs":[],"block_height":114580308,"block_hash":"9fqHkn3WQMycDGXkZ3P21oym5o9uuUeDeSqFgs1mXEGP"},"id":"dontcare"}

Version (please complete the following information):

• nearcore commit/branch: neard (release 1.39.1) (build 1.39.1) (rustc 1.75.0) (protocol 66) (db 38)
• rust version (if local): NA
• docker (if using docker): We're not using docker
• mainnet/testnet/betanet/local: mainnet

Additional context

• We are able to get responses from other requests with the query method.

• The snapshots we downloadeded are

  • cold-data: near_cf://near-protocol-public/backups/mainnet/archive/2024-04-14T23:07:30Z/cold-data
  • hot-data: near_cf://near-protocol-public/backups/mainnet/archive/2024-04-16T23:07:27Z/hot-data

• I saw that in other issue indicating that MissingTrieValue might indicate data corruption. The block 114580308 is settled on (Mar-12-2024 6:12:43 PM) +UTC so there might be corrupted data in the archive server used to generate the official snapshots.

[Longarithm]

From what I know, this is known issue and @posvyatokum is working on it. This one is quite insidious and we don't have clear ETA.

[qnduncan]

Thanks for the reply.

[posvyatokum]

@qnduncan Another thing to add, is that archival-rpc.mainnet.near.org service operates using Read RPC. Because of this, they are able to fulfil historical requests better.

[quick-pawiromitchel]

Hi team, is there any ETA on this fix? We're running into the same issues with this request

{
  "jsonrpc": "2.0",
  "method": "query",
  "id": 13569,
  "params": {
    "request_type": "call_function",
    "account_id": "consensus_finoa_00.poolv1.near",
    "method_name": "get_account",
    "args_base64": "eyJhY2NvdW50X2lkIjoicmVzdGFrZS5uZWFyIn0=",
    "block_id": 120308219
  }
}

I ran this against our nodes and they all return

"error_message": "StorageError(MissingTrieValue(TrieStorage, DVM721HHV7A6gUdUVv8Zy7qzKborJUwCqR12deHSjbni))"

Tried to run this against the public RPC as well archival-rpc.mainnet.near.org but it also returns an error

{
  "jsonrpc": "2.0",
  "error": {
    "code": -32000,
    "message": "Server error",
    "data": "Failed to serialize JsonRpcError",
    "name": "INTERNAL_ERROR",
    "cause": {
      "info": {
        "error_message": "Failed to serialize JsonRpcError"
      },
      "name": "INTERNAL_ERROR"
    }
  },
  "id": 13569
}

[quick-pawiromitchel]

bump

[tbuckley-bd]

Hello, is there any update on this? We are also running into this issue against multiple get_accounts requests at multiple blocks, here is an example of one

{
  "method": "query",
  "params": {
    "block_id": 124343511,
    "request_type": "call_function",
    "account_id": "zavodil.poolv1.near",
    "method_name": "get_accounts",
    "args_base64": "eyJmcm9tX2luZGV4IjogMCwibGltaXQiOiAyMDB9"
  },
  "id": "1",
  "jsonrpc": "2.0"
}

where we see both the MissingTrieValue and Failed to serialize JsonRpcError errors

[telezhnaya]

Could you please check if it's still an issue?

[telezhnaya]

Closing this as a duplicate of https://github.com/near/nearcore/issues/11174

[ella-quicknode]

looked like this is solved now. thanks