hyperledger / besu

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client https://wiki.hyperledger.org/display/besu
https://www.hyperledger.org/projects/besu
Apache License 2.0
1.48k stars 803 forks source link

State rolling failed on BonsaiWorldStateKeyValueStorage because nonces differ #6032

Open iampkuhz opened 11 months ago

iampkuhz commented 11 months ago

Description

As an [Actor], I want [feature] so that [why].

Acceptance Criteria

Steps to Reproduce (Bug)

  1. start the first 3 nodes.
  2. start the 4th nodes and it begin syncing

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Frequency: [What percentage of the time does it occur?] I've tried restart the node but it still stucked.

Logs (if a bug)

2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278438 (0x24410b77242b34625a176865f9e28e5a7a5c205156a279b0aa7275d0432a519c), - Mg/s, Peers: 1
2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278439 (0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e), - Mg/s, Peers: 1
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | Attempting Rollback of 0x533a4388377b211e995ad2f4a317cf1f446240496ceadc3446756fc2e0c25cdb
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | State rolling failed on BonsaiWorldStateKeyValueStorage for block hash 0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e java.lang.IllegalStateException: Address=0x1a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7 Prior Value in Rolling Change: nonces differ
        at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.assertCloseEnoughForDiffing(BonsaiAccount.java:313)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollAccountChange(BonsaiWorldStateUpdateAccumulator.java:577)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.lambda$rollBack$15(BonsaiWorldStateUpdateAccumulator.java:526)
        at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollBack(BonsaiWorldStateUpdateAccumulator.java:524)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.rollMutableStateToBlockHash(BonsaiWorldStateProvider.java:242)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:184)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:160)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:124)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:72)
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:45)
        at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:45)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:63)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | MainnetBlockValidator | Invalid block 278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf): Optional[Unable to process block because parent world state 0x0bbc969cf30f94ebf0ff9a38fc0d8858a9553714308c79ed4278ca2fc00ede8b is not available]
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | Pipeline | Unexpected exception in pipeline. Aborting. org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block at #278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:65)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

Versions (Add all that apply)

non-fungible-nelson commented 11 months ago

Can you provide us with your configuration and genesis files?

Any other details that may be relevant.

iampkuhz commented 11 months ago
  1. here is my config file.

    BTW, if I change data-storage-format from BONSAI to FOREST and clean data to re-sync, the sync works quite normal

    
    data-path="/home/admin/node4/data" # Path

node-private-key-file="/home/admin/node4/conf/key"

Network

bootnodes=["--(hidden)--"]

可以访问服务器的客户端IP白名单

host-allowlist=["*"]

如果客户端使用浏览器访问,可以

rpc-http-cors-origins=["all"]

p2p-host="--(hidden)--" p2p-port=30306 max-peers=42

HTTP JSON-RPC

rpc-http-enabled=true rpc-http-host="0.0.0.0" rpc-http-port=8548 rpc-http-api=["ETH","NET","QBFT", "DEBUG"]

permissions-nodes-config-file-enabled=true permissions-nodes-config-file="/home/admin/node4/conf/permissions_config.toml" static-nodes-file="/home/admin/node4/conf/static-nodes.json"

WS JSON-RPC

rpc-ws-host="0.0.0.0" rpc-ws-port=8604

Chain

genesis-file="/home/admin/node4/conf/genesis.json" # Path to the custom genesis file

关闭发现模式

discovery-enabled=true nat-method="NONE"

data storage format

data-storage-format="BONSAI"

sync-mode="FULL"

fast-sync-min-peers=3

tx-pool-limit-by-account-percentage=1

节点内存中缓存待出块的交易数量最大值

tx-pool-max-size="2000"

Mining

miner-enabled=true

miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"


2. here is my genesis file
```json
{
  "config" : {
    "chainId" : 20230824001,
    "berlinBlock" : 0,
    "qbft" : {
      "blockperiodseconds" : 4,
      "epochlength" : 30000,
      "requesttimeoutseconds" : 8
    }
  },
  "nonce" : "0x0",
  "timestamp" : "0x58ee40ba",
  "gasLimit" : "0x1fffffffffffff",
  "contractSizeLimit": 2147483647,
  "difficulty" : "0x1",
  "mixHash" : "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "coinbase" : "0x0000000000000000000000000000000000000000",
  "alloc" : {
    "3355ff1765cea77300d79b0821c10d69bbce4029" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "0xad78ebc5ac6200000"
    },
    "5ee46326f4ee40e388730e63b9aeecaddc573771" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "fab25c4214ebab8f02563d1ab821f2fa337fcdb8" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "e0dbb7c3f8b8f875f01e277a7364fee98e252625" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    }
  },
  "extraData" : "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f854948ea92c56f8157187d8441babefa146701c414fe794d94f00cc3896283e21b7481759adcc57bf0d7205941a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7946ed28d2fd48ef53a7fe39f65c14a86ceccdfb457c080c0"
}
non-fungible-nelson commented 10 months ago

@matthew1001 any insight into this scenario? More fodder for Bonsai --> QBFT testing...

matthew1001 commented 10 months ago

I've not done much with Bonsai yet, only Forest, so it's not a scenario I've tried myself. Does FULL sync request all transactions from block 0 (I think it does?) and if so, can Bonsai nodes provide that?