Unable to recover blocks after worldstate database issue

[aggcastro]

DESCRIPTION

Hello, I am having a world state database issue on version 24.5.2 in a private network with 6 nodes using QBFT and Bonsai as storage format.

Here are the logs:

{"time":"2024-07-17T21:00:14.051229504Z","stream":"stdout","_p":"F","log":"org.hyperledger.besu.ethereum.trie.MerkleTrieException: **Unable to load trie node value for hash** 0x46228703f771892700b8e0b22211b4d93644eed8c8130c9b6101e7340a420ddd location 0x","kubernetes":{"pod_name":"besu-node-validator-3-0","namespace_name":"besu","pod_id":"0b89b60d-af9b-45e1-8363-61f942bfa225","labels":{"app.kubernetes.io/component":"besu","app.kubernetes.io/managed-by":"helm","app.kubernetes.io/name":"besu-statefulset","app.kubernetes.io/namespace":"besu","app.kubernetes.io/part-of":"besu-node-validator-3","app.kubernetes.io/release":"besu-validator-3","apps.kubernetes.io/pod-index":"0","controller-revision-hash":"besu-node-validator-3-7bf6b669df","statefulset.kubernetes.io/pod-name":"besu-node-validator-3-0"},"annotations":{"prometheus.io/path":"/metrics","prometheus.io/port":"9545","prometheus.io/scrape":"true"},"host":"ip-10-0-6-15.eu-west-1.compute.internal","container_name":"besu-validator-3-besu","docker_id":"ffd5a63206a790faaf38af30b91093646e2795c10d73c82877dc3df2e5303873","container_hash":"docker.io/hyperledger/besu@sha256:db39870bfa72d4aa02f39268211d8037fda4fc81ae4cb7a06e024409fa5fb55b","container_image":"docker.io/hyperledger/besu:24.5.2"}}

VERSIONS (Add all that apply)

• OpenJDK:

openjdk 21.0.2 2024-01-16 OpenJDK Runtime Environment (build 21.0.2+13-Ubuntu-123.10.1) OpenJDK 64-Bit Server VM (build 21.0.2+13-Ubuntu-123.10.1, mixed mode, sharing) besu/v24.5.2/linux-x86_64/openjdk-java-21

• Kernel version:

5.10.216-204.855.amzn2.x86_64

• SO:

PRETTY_NAME="Ubuntu 23.10" NAME="Ubuntu" VERSION_ID="23.10" VERSION="23.10 (Mantic Minotaur)" VERSION_CODENAME=mantic ID=ubuntu ID_LIKE=debian HOME_URL=https://www.ubuntu.com/ SUPPORT_URL=https://help.ubuntu.com/ BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy UBUNTU_CODENAME=mantic LOGO=ubuntu-logo

Additionally, the node is deployed in a k8s cluster on AWS via a Docker container.

Thanks in advance.

[non-fungible-nelson]

hi there - can you provide your complete configuration?

@garyschulte this looks like a strange one "at location 0x"

@matthew1001 for visibility.

[siladu]

If it could provide more logs that would be helpful thanks, particular the logs leading up to when the issue began.