search

hyperledger / besu

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client https://wiki.hyperledger.org/display/besu
https://www.hyperledger.org/projects/besu
Apache License 2.0
1.52k stars 840 forks source link

State rolling failed on BonsaiWorldStateKeyValueStorage because nonces differ #6032

Open iampkuhz opened 1 year ago

iampkuhz commented 1 year ago

Description

As an [Actor], I want [feature] so that [why].

Acceptance Criteria

Steps to Reproduce (Bug)

  1. start the first 3 nodes.
  2. start the 4th nodes and it begin syncing

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Frequency: [What percentage of the time does it occur?] I've tried restart the node but it still stucked.

Logs (if a bug)

2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278438 (0x24410b77242b34625a176865f9e28e5a7a5c205156a279b0aa7275d0432a519c), - Mg/s, Peers: 1
2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278439 (0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e), - Mg/s, Peers: 1
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | Attempting Rollback of 0x533a4388377b211e995ad2f4a317cf1f446240496ceadc3446756fc2e0c25cdb
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | State rolling failed on BonsaiWorldStateKeyValueStorage for block hash 0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e java.lang.IllegalStateException: Address=0x1a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7 Prior Value in Rolling Change: nonces differ
        at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.assertCloseEnoughForDiffing(BonsaiAccount.java:313)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollAccountChange(BonsaiWorldStateUpdateAccumulator.java:577)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.lambda$rollBack$15(BonsaiWorldStateUpdateAccumulator.java:526)
        at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollBack(BonsaiWorldStateUpdateAccumulator.java:524)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.rollMutableStateToBlockHash(BonsaiWorldStateProvider.java:242)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:184)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:160)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:124)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:72)
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:45)
        at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:45)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:63)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | MainnetBlockValidator | Invalid block 278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf): Optional[Unable to process block because parent world state 0x0bbc969cf30f94ebf0ff9a38fc0d8858a9553714308c79ed4278ca2fc00ede8b is not available]
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | Pipeline | Unexpected exception in pipeline. Aborting. org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block at #278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:65)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

Versions (Add all that apply)

non-fungible-nelson commented 1 year ago

Can you provide us with your configuration and genesis files?

Any other details that may be relevant.

iampkuhz commented 1 year ago
  1. here is my config file.

    BTW, if I change data-storage-format from BONSAI to FOREST and clean data to re-sync, the sync works quite normal

    
data-path="/home/admin/node4/data" # Path

node-private-key-file="/home/admin/node4/conf/key"

Network

bootnodes=["--(hidden)--"]

可以访问服务器的客户端IP白名单

host-allowlist=["*"]

如果客户端使用浏览器访问,可以

rpc-http-cors-origins=["all"]

p2p-host="--(hidden)--" p2p-port=30306 max-peers=42

HTTP JSON-RPC

rpc-http-enabled=true rpc-http-host="0.0.0.0" rpc-http-port=8548 rpc-http-api=["ETH","NET","QBFT", "DEBUG"]

permissions-nodes-config-file-enabled=true permissions-nodes-config-file="/home/admin/node4/conf/permissions_config.toml" static-nodes-file="/home/admin/node4/conf/static-nodes.json"

WS JSON-RPC

rpc-ws-host="0.0.0.0" rpc-ws-port=8604

Chain

genesis-file="/home/admin/node4/conf/genesis.json" # Path to the custom genesis file

关闭发现模式

discovery-enabled=true nat-method="NONE"

data storage format

data-storage-format="BONSAI"

sync-mode="FULL"

fast-sync-min-peers=3

tx-pool-limit-by-account-percentage=1

节点内存中缓存待出块的交易数量最大值

tx-pool-max-size="2000"

Mining

miner-enabled=true

miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"


2. here is my genesis file
```json
{
  "config" : {
    "chainId" : 20230824001,
    "berlinBlock" : 0,
    "qbft" : {
      "blockperiodseconds" : 4,
      "epochlength" : 30000,
      "requesttimeoutseconds" : 8
    }
  },
  "nonce" : "0x0",
  "timestamp" : "0x58ee40ba",
  "gasLimit" : "0x1fffffffffffff",
  "contractSizeLimit": 2147483647,
  "difficulty" : "0x1",
  "mixHash" : "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "coinbase" : "0x0000000000000000000000000000000000000000",
  "alloc" : {
    "3355ff1765cea77300d79b0821c10d69bbce4029" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "0xad78ebc5ac6200000"
    },
    "5ee46326f4ee40e388730e63b9aeecaddc573771" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "fab25c4214ebab8f02563d1ab821f2fa337fcdb8" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "e0dbb7c3f8b8f875f01e277a7364fee98e252625" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    }
  },
  "extraData" : "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f854948ea92c56f8157187d8441babefa146701c414fe794d94f00cc3896283e21b7481759adcc57bf0d7205941a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7946ed28d2fd48ef53a7fe39f65c14a86ceccdfb457c080c0"
}
non-fungible-nelson commented 1 year ago

@matthew1001 any insight into this scenario? More fodder for Bonsai --> QBFT testing...

matthew1001 commented 1 year ago

I've not done much with Bonsai yet, only Forest, so it's not a scenario I've tried myself. Does FULL sync request all transactions from block 0 (I think it does?) and if so, can Bonsai nodes provide that?