search

node-real / bsc-erigon

Ethereum implementation on the efficiency frontier
GNU Lesser General Public License v3.0
123 stars 40 forks source link

1.2.12 hotfix wrong trie root #430

Closed temuera closed 1 month ago

temuera commented 2 months ago

How to fix broken data. Try fix follow those step, one by one, just need unwind 1 block, use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind.

  1. make integration && make erigon
  2. ./build/bin/integration state_stages --unwind=1 --chain=bsc --datadir=./data
  3. ./build/bin/integration stage_headers --unwind=1 --chain=bsc --datadir=./data
  4. add --stage.step=1flag to start command and then restart.

If you node back to tip, just stop and switch to the latest main branch, and restart as before.

WangXiaopai595 commented 1 month ago

Make integration use this https://github.com/blxdyx/bsc-erigon/tree/temp_fix

/home/erigon/bsc-erigon-temp_fix/build/bin/integration state_stages --unwind=20 --chain=bsc --datadir=./data
INFO[07-11|10:21:42.116] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
mdbx_setup_dxb:15946 filesize mismatch (expect 8792955682816b/536679424p, have 8793844875264b/536733696p)
INFO[07-11|10:21:42.129] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|10:21:46.688] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.3GB sys=2.4GB
INFO[07-11|10:21:46.692] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[07-11|10:21:46.720] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|10:21:46.728] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
INFO[07-11|10:21:49.151] [10/15 CallTraces] Unwind                from=40343181 to=40343161
INFO[07-11|10:21:49.566] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=true
INFO[07-11|10:21:49.609] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=false
INFO[07-11|10:21:49.746] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=true codes=false
INFO[07-11|10:21:50.067] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=AccountChangeSet
INFO[07-11|10:21:50.134] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=StorageChangeSet
EROR[07-11|10:21:51.373] [9/15 IntermediateHashes] block Number 40343161, block hash 0xadd61ef1d94009f180c11f329ee0379c53e49d862c36439129f1d8820f6a7e2c, err = wrong trie root: 5ad231002c0f58070379de4ab617b7f0d4bb2f745217060f10b7e65b9eae1816, expected (from header): 7ffd370e413891d2e0d6e43d7d3a2aef0cf86772eafab24271b2f7d9c83ce882

Could you show me the logs about the node. Main about when insert the wrong 40343181 block. Error starting here

[INFO] [07-11|10:37:45.647] [5/15 Bodies] Processed                  highest=40367552 blocks=24371 in=369.729 blk/sec=65
[INFO] [07-11|10:37:45.647] [5/15 Bodies] DONE                       in=6m9.72883019s
[INFO] [07-11|10:37:45.647] [6/15 Senders] Started                   from=40343181 to=40367552
[DBUG] [07-11|10:37:47.203] [downloader] new anchor too far in the past: 0, latest header in db: 40367552 
[DBUG] [07-11|10:37:53.425] [6/15 Senders] DONE                      in=7.777779242s
[INFO] [07-11|10:37:53.436] [7/15 Execution] Blocks execution        from=40343181 to=40367552
[WARN] [07-11|10:37:53.447] [7/15 Execution] Execution failed        block=40343182 hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err="invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537"
[DBUG] [07-11|10:37:53.447] UnwindTo                                 block=40343181 block_hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err="invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537"
[INFO] [07-11|10:37:53.448] [7/15 Execution] Completed on            block=40343181
[DBUG] [07-11|10:37:53.448] [7/15 Execution] DONE                    in=23.006118ms
[WARN] [07-11|10:37:56.343] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[INFO] [07-11|10:37:56.343] Timings (slower than 50ms)               Snapshots=4.452s Headers=1.819s BlockHashes=280ms Bodies=6m9.728s Senders=7.777s Unwind Bodies=1.152s Unwind Headers=1.74s
blxdyx commented 1 month ago

Make integration use this https://github.com/blxdyx/bsc-erigon/tree/temp_fix

/home/erigon/bsc-erigon-temp_fix/build/bin/integration state_stages --unwind=20 --chain=bsc --datadir=./data
INFO[07-11|10:21:42.116] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
mdbx_setup_dxb:15946 filesize mismatch (expect 8792955682816b/536679424p, have 8793844875264b/536733696p)
INFO[07-11|10:21:42.129] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|10:21:46.688] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.3GB sys=2.4GB
INFO[07-11|10:21:46.692] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[07-11|10:21:46.720] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|10:21:46.728] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
INFO[07-11|10:21:49.151] [10/15 CallTraces] Unwind                from=40343181 to=40343161
INFO[07-11|10:21:49.566] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=true
INFO[07-11|10:21:49.609] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=false
INFO[07-11|10:21:49.746] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=true codes=false
INFO[07-11|10:21:50.067] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=AccountChangeSet
INFO[07-11|10:21:50.134] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=StorageChangeSet
EROR[07-11|10:21:51.373] [9/15 IntermediateHashes] block Number 40343161, block hash 0xadd61ef1d94009f180c11f329ee0379c53e49d862c36439129f1d8820f6a7e2c, err = wrong trie root: 5ad231002c0f58070379de4ab617b7f0d4bb2f745217060f10b7e65b9eae1816, expected (from header): 7ffd370e413891d2e0d6e43d7d3a2aef0cf86772eafab24271b2f7d9c83ce882

Could you show me the logs about the node. Main about when insert the wrong 40343181 block. Error starting here

[INFO] [07-11|10:37:45.647] [5/15 Bodies] Processed                  highest=40367552 blocks=24371 in=369.729 blk/sec=65
[INFO] [07-11|10:37:45.647] [5/15 Bodies] DONE                       in=6m9.72883019s
[INFO] [07-11|10:37:45.647] [6/15 Senders] Started                   from=40343181 to=40367552
[DBUG] [07-11|10:37:47.203] [downloader] new anchor too far in the past: 0, latest header in db: 40367552 
[DBUG] [07-11|10:37:53.425] [6/15 Senders] DONE                      in=7.777779242s
[INFO] [07-11|10:37:53.436] [7/15 Execution] Blocks execution        from=40343181 to=40367552
[WARN] [07-11|10:37:53.447] [7/15 Execution] Execution failed        block=40343182 hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err="invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537"
[DBUG] [07-11|10:37:53.447] UnwindTo                                 block=40343181 block_hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err="invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537"
[INFO] [07-11|10:37:53.448] [7/15 Execution] Completed on            block=40343181
[DBUG] [07-11|10:37:53.448] [7/15 Execution] DONE                    in=23.006118ms
[WARN] [07-11|10:37:56.343] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[INFO] [07-11|10:37:56.343] Timings (slower than 50ms)               Snapshots=4.452s Headers=1.819s BlockHashes=280ms Bodies=6m9.728s Senders=7.777s Unwind Bodies=1.152s Unwind Headers=1.74s

Before you restart, it should have insert wrong block 40343181. Show about this.

WangXiaopai595 commented 1 month ago

Before you restart, it should have insert wrong block 40343181. Show about this.

How should I operate it

blxdyx commented 1 month ago

Before you restart, it should have insert wrong block 40343181. Show about this.

How should I operate it

Just logs before you restart

WangXiaopai595 commented 1 month ago

Before you restart, it should have insert wrong block 40343181. Show about this.

How should I operate it

Just logs before you restart

I started using Docker before, but the logs have been lost

WangXiaopai595 commented 1 month ago

I started using Docker before, but the logs have been lost

This is the earliest log I have kept

{"log":"[INFO] [07-10|07:42:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.7GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:42:31.947942351Z"}
{"log":"[INFO] [07-10|07:43:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.8GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:43:01.94828717Z"}
{"log":"[INFO] [07-10|07:43:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.9GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:43:31.948124571Z"}
{"log":"[INFO] [07-10|07:44:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.4GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:44:01.94802962Z"}
{"log":"[INFO] [07-10|07:44:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:44:31.948610171Z"}
{"log":"[INFO] [07-10|07:44:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:44:31.948660174Z"}
{"log":"[INFO] [07-10|07:44:57.566] [p2p] GoodPeers                          eth68=1\n","stream":"stderr","time":"2024-07-10T07:44:57.566390969Z"}
{"log":"[INFO] [07-10|07:44:57.648] [mem] memory stats                       Rss=3.5GB Size=15.5TB Pss=3.5GB SharedClean=30.8MB SharedDirty=0B PrivateClean=302.3MB PrivateDirty=3.2GB Referenced=3.5GB Anonymous=3.2GB Swap=0B alloc=2.5GB sys=3.6GB\n","stream":"stderr","time":"2024-07-10T07:44:57.648502485Z"}
{"log":"[INFO] [07-10|07:45:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:45:01.948624027Z"}
{"log":"[INFO] [07-10|07:45:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:45:01.948663228Z"}
{"log":"[INFO] [07-10|07:45:31.949] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:45:31.949790975Z"}
{"log":"[INFO] [07-10|07:45:31.949] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:45:31.949849426Z"}
{"log":"[INFO] [07-10|07:46:01.949] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:46:01.950300869Z"}
{"log":"[INFO] [07-10|07:46:01.950] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:46:01.9503403Z"}
{"log":"[INFO] [07-10|07:46:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:46:31.948000394Z"}
{"log":"[INFO] [07-10|07:46:31.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:46:31.948052637Z"}
{"log":"[INFO] [07-10|07:47:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:47:01.949020475Z"}
{"log":"[INFO] [07-10|07:47:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=24 skelMin=40343180 skelMax=40380236 resp=24 respMin=40343180 respMax=40343180 dups=24\n","stream":"stderr","time":"2024-07-10T07:47:01.949068161Z"}
{"log":"[INFO] [07-10|07:47:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.7GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:47:31.947743194Z"}
{"log":"[INFO] [07-10|07:47:31.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=24 respMin=40343180 respMax=40343180 dups=24\n","stream":"stderr","time":"2024-07-10T07:47:31.947793157Z"}
{"log":"[INFO] [07-10|07:47:57.565] [p2p] GoodPeers                          eth68=1\n","stream":"stderr","time":"2024-07-10T07:47:57.565871109Z"}
{"log":"[INFO] [07-10|07:47:57.651] [mem] memory stats                       Rss=3.5GB Size=15.5TB Pss=3.5GB SharedClean=30.8MB SharedDirty=0B PrivateClean=332.3MB PrivateDirty=3.2GB Referenced=3.5GB Anonymous=3.2GB Swap=0B alloc=2.8GB sys=3.6GB\n","stream":"stderr","time":"2024-07-10T07:47:57.651922912Z"}
{"log":"[INFO] [07-10|07:48:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.8GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:48:01.948001782Z"}
{"log":"[INFO] [07-10|07:48:01.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:48:01.948046105Z"}
{"log":"[INFO] [07-10|07:48:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:48:31.948801796Z"}
{"log":"[INFO] [07-10|07:48:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=26 respMin=40343180 respMax=40343180 dups=26\n","stream":"stderr","time":"2024-07-10T07:48:31.948850102Z"}
{"log":"[INFO] [07-10|07:49:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.9GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:49:01.947999766Z"}
{"log":"[INFO] [07-10|07:49:01.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:49:01.9480647Z"}
{"log":"[INFO] [07-10|07:49:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=3.3GB sys=3.6GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:49:31.949221382Z"}
{"log":"[INFO] [07-10|07:49:31.949] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:49:31.949272854Z"}
{"log":"[INFO] [07-10|07:50:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=3.6GB sys=3.9GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:50:01.948774582Z"}
{"log":"[INFO] [07-10|07:50:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:50:01.948821519Z"}
{"log":"[INFO] [07-10|07:50:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.8GB sys=3.9GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:50:31.948385664Z"}
{"log":"[INFO] [07-10|07:50:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=24 respMin=40343180 respMax=40343180 dups=24\n","stream":"stderr","time":"2024-07-10T07:50:31.948544788Z"}
{"log":"[INFO] [07-10|07:50:57.566] [p2p] GoodPeers                          eth68=1\n","stream":"stderr","time":"2024-07-10T07:50:57.566672249Z"}
{"log":"[INFO] [07-10|07:50:57.671] [mem] memory stats                       Rss=4.6GB Size=15.5TB Pss=4.6GB SharedClean=30.8MB SharedDirty=0B PrivateClean=782.5MB PrivateDirty=3.8GB Referenced=4.6GB Anonymous=3.8GB Swap=0B alloc=3.1GB sys=3.9GB\n","stream":"stderr","time":"2024-07-10T07:50:57.672056582Z"}
{"log":"[INFO] [07-10|07:51:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=3.2GB sys=3.9GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:51:01.947566397Z"}
{"log":"[INFO] [07-10|07:51:01.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:51:01.947607359Z"}
{"log":"[INFO] [07-10|07:51:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=3.5GB sys=3.9GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:51:31.948229293Z"}
{"log":"[INFO] [07-10|07:51:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:51:31.949001384Z"}
{"log":"[INFO] [07-10|07:52:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=3.8GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:52:01.948613389Z"}
{"log":"[INFO] [07-10|07:52:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:52:01.948661635Z"}
{"log":"[INFO] [07-10|07:52:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:52:31.948309005Z"}
{"log":"[INFO] [07-10|07:52:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=26 respMin=40343180 respMax=40343180 dups=26\n","stream":"stderr","time":"2024-07-10T07:52:31.948358676Z"}
{"log":"[INFO] [07-10|07:53:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:53:01.947978167Z"}
{"log":"[INFO] [07-10|07:53:01.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:53:01.948026288Z"}
{"log":"[INFO] [07-10|07:53:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:53:31.947695606Z"}
{"log":"[INFO] [07-10|07:53:31.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:53:31.947737086Z"}
{"log":"[INFO] [07-10|07:53:57.566] [p2p] GoodPeers                          eth68=1\n","stream":"stderr","time":"2024-07-10T07:53:57.566853505Z"}
{"log":"[INFO] [07-10|07:53:57.674] [mem] memory stats                       Rss=5.0GB Size=15.5TB Pss=5.0GB SharedClean=30.8MB SharedDirty=0B PrivateClean=947.6MB PrivateDirty=4.0GB Referenced=5.0GB Anonymous=4.0GB Swap=0B alloc=2.7GB sys=4.1GB\n","stream":"stderr","time":"2024-07-10T07:53:57.675176266Z"}
{"log":"[INFO] [07-10|07:54:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.7GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:54:01.948518335Z"}
{"log":"[INFO] [07-10|07:54:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:54:01.948553685Z"}
{"log":"[INFO] [07-10|07:54:31.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:54:31.948571746Z"}
{"log":"[INFO] [07-10|07:54:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:54:31.948614166Z"}
{"log":"[INFO] [07-10|07:55:01.948] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:55:01.948284Z"}
{"log":"[INFO] [07-10|07:55:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:55:01.948323463Z"}
{"log":"[INFO] [07-10|07:55:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.6GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:55:31.947816078Z"}
{"log":"[INFO] [07-10|07:55:31.947] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:55:31.947858195Z"}
{"log":"[INFO] [07-10|07:56:01.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.7GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:56:01.948584138Z"}
{"log":"[INFO] [07-10|07:56:01.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:56:01.948625838Z"}
{"log":"[INFO] [07-10|07:56:31.947] [2/15 Headers] No block headers to write in this log period number=40343181 blk/second=0.000 alloc=2.5GB sys=4.1GB invalidHeaders=0 rejectedBadHeaders=0\n","stream":"stderr","time":"2024-07-10T07:56:31.948266351Z"}
{"log":"[INFO] [07-10|07:56:31.948] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=25 skelMin=40343180 skelMax=40380236 resp=25 respMin=40343180 respMax=40343180 dups=25\n","stream":"stderr","time":"2024-07-10T07:56:31.948310875Z"}
{"log":"[INFO] [07-10|07:56:44.721] Got interrupt, shutting down... \n","stream":"stderr","time":"2024-07-10T07:56:44.721704208Z"}
{"log":"[INFO] [07-10|07:56:44.721] Got interrupt, shutting down...          sig=terminated\n","stream":"stderr","time":"2024-07-10T07:56:44.721755898Z"}
{"log":"[INFO] [07-10|07:56:44.721] Exiting Engine... \n","stream":"stderr","time":"2024-07-10T07:56:44.721843315Z"}
{"log":"[INFO] [07-10|07:56:44.721] Exiting... \n","stream":"stderr","time":"2024-07-10T07:56:44.721875309Z"}
{"log":"[INFO] [07-10|07:56:44.721] RPC server shutting down \n","stream":"stderr","time":"2024-07-10T07:56:44.721881906Z"}
{"log":"[INFO] [07-10|07:56:44.721] RPC server shutting down \n","stream":"stderr","time":"2024-07-10T07:56:44.721897026Z"}
{"log":"[INFO] [07-10|07:56:44.721] HTTP endpoint closed                     url=127.0.0.1:8545\n","stream":"stderr","time":"2024-07-10T07:56:44.721950909Z"}
{"log":"[INFO] [07-10|07:56:44.721] Engine HTTP endpoint close               url=127.0.0.1:8551\n","stream":"stderr","time":"2024-07-10T07:56:44.721958466Z"}
{"log":"[INFO] [07-10|07:56:44.721] RPC server shutting down \n","stream":"stderr","time":"2024-07-10T07:56:44.721964536Z"}
{"log":"[INFO] [07-10|08:03:32.204] logging to file system                   log dir=/data/log file prefix=erigon log level=info json=false\n","stream":"stderr","time":"2024-07-10T08:03:32.205041584Z"}
{"log":"[INFO] [07-10|08:03:32.205] Enabling metrics export to prometheus    path=http://0.0.0.0:6060/debug/metrics/prometheus\n","stream":"stderr","time":"2024-07-10T08:03:32.205451773Z"}
{"log":"[INFO] [07-10|08:03:32.205] Build info                               git_branch= git_tag= git_commit=\n","stream":"stderr","time":"2024-07-10T08:03:32.205479333Z"}
{"log":"[INFO] [07-10|08:03:32.205] Starting Erigon on BSC mainnet... \n","stream":"stderr","time":"2024-07-10T08:03:32.20549108Z"}
{"log":"[INFO] [07-10|08:03:32.206] Maximum peer count                       ETH=1000 total=1000\n","stream":"stderr","time":"2024-07-10T08:03:32.207190636Z"}
{"log":"[INFO] [07-10|08:03:32.207] starting HTTP APIs                       port=8545 APIs=eth,erigon,engine\n","stream":"stderr","time":"2024-07-10T08:03:32.20735482Z"}
{"log":"[INFO] [07-10|08:03:32.207] torrent verbosity                        level=WRN\n","stream":"stderr","time":"2024-07-10T08:03:32.207557404Z"}
{"log":"[INFO] [07-10|08:03:34.214] Set global gas cap                       cap=50000000\n","stream":"stderr","time":"2024-07-10T08:03:34.214782463Z"}
{"log":"[INFO] [07-10|08:03:34.227] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb\n","stream":"stderr","time":"2024-07-10T08:03:34.227249378Z"}
{"log":"[INFO] [07-10|08:03:34.227] Opening Database                         label=chaindata path=/data/chaindata\n","stream":"stderr","time":"2024-07-10T08:03:34.227341338Z"}
{"log":"[INFO] [07-10|08:03:34.230] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384\n","stream":"stderr","time":"2024-07-10T08:03:34.231146755Z"}
{"log":"[INFO] [07-10|08:03:34.254] Initialised chain configuration          config=\"{ChainID: 56 Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024, Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime 1705996800, FeynmanTime 1713419340, FeynmanFixTime 1713419340, CancunTime 1718863500, HaberTime 1718863500, HaberFixTime 1720591588, Engine: parlia}\" genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b\n","stream":"stderr","time":"2024-07-10T08:03:34.254821458Z"}
{"log":"[INFO] [07-10|08:04:00.401] Initialising Ethereum protocol           network=56\n","stream":"stderr","time":"2024-07-10T08:04:00.40227808Z"}
{"log":"[INFO] [07-10|08:04:00.402] Opening Database                         label=parlia path=/data/parlia\n","stream":"stderr","time":"2024-07-10T08:04:00.402319283Z"}
{"log":"mdbx_setup_dxb:15946 filesize mismatch (expect 134217728b/32768p, have 132608000b/32375p)\n","stream":"stderr","time":"2024-07-10T08:04:00.40865267Z"}
{"log":"mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(93ca869dbcaf1161-f02183534d5f4563) is MATCH: rollback NOT needed, steady-sync NEEDED\n","stream":"stderr","time":"2024-07-10T08:04:00.409834987Z"}
{"log":"[INFO] [07-10|08:04:00.410] Opening Database                         label=blob path=/data/blobs/blob\n","stream":"stderr","time":"2024-07-10T08:04:00.410737068Z"}
{"log":"[INFO] [07-10|08:04:01.194] Starting private RPC server              on=localhost:9090\n","stream":"stderr","time":"2024-07-10T08:04:01.195026891Z"}
{"log":"[INFO] [07-10|08:04:01.194] new subscription to logs established \n","stream":"stderr","time":"2024-07-10T08:04:01.195076278Z"}
{"log":"[INFO] [07-10|08:04:01.195] rpc filters: subscribing to Erigon events \n","stream":"stderr","time":"2024-07-10T08:04:01.195570008Z"}
{"log":"[INFO] [07-10|08:04:01.195] new subscription to newHeaders established \n","stream":"stderr","time":"2024-07-10T08:04:01.19567311Z"}
{"log":"[INFO] [07-10|08:04:01.196] Reading JWT secret                       path=/data/jwt.hex\n","stream":"stderr","time":"2024-07-10T08:04:01.19715324Z"}
{"log":"[INFO] [07-10|08:04:01.197] JsonRpc endpoint opened                  ws=false ws.compression=true grpc=false http.url=127.0.0.1:8545\n","stream":"stderr","time":"2024-07-10T08:04:01.197573235Z"}
{"log":"[INFO] [07-10|08:04:01.197] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true\n","stream":"stderr","time":"2024-07-10T08:04:01.197606232Z"}
{"log":"[WARN] [07-10|08:04:01.212] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. \n","stream":"stderr","time":"2024-07-10T08:04:01.212932059Z"}
{"log":"[WARN] [07-10|08:04:01.212] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. \n","stream":"stderr","time":"2024-07-10T08:04:01.212991279Z"}
{"log":"[WARN] [07-10|08:04:01.212] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. \n","stream":"stderr","time":"2024-07-10T08:04:01.213035763Z"}
{"log":"[WARN] [07-10|08:04:01.212] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. \n","stream":"stderr","time":"2024-07-10T08:04:01.213064029Z"}
{"log":"[WARN] [07-10|08:04:01.217] NAT ExternalIP resolution has failed, try to pass a different --nat option err=\"no UPnP or NAT-PMP router discovered\"\n","stream":"stderr","time":"2024-07-10T08:04:01.217506854Z"}
{"log":"[INFO] [07-10|08:04:01.220] Started P2P networking                   version=68 self=enode://f1284c6bcf578c43e1439b78fcc5239f21c5197458369693e3b1bb3f8a7697049101e2882efcd3049962ca5d9cac9f1eebe401241ad38568bca96b81ba7a7899@127.0.0.1:30311 name=erigon/v1.2.11/linux-amd64/go1.21.11\n","stream":"stderr","time":"2024-07-10T08:04:01.220664891Z"}
{"log":"[INFO] [07-10|08:04:01.251] [1/15 Snapshots] Requesting downloads \n","stream":"stderr","time":"2024-07-10T08:04:01.252097581Z"}
{"log":"[WARN] [07-10|08:04:04.379] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=29499999 min_in_db=29704600\n","stream":"stderr","time":"2024-07-10T08:04:04.379483154Z"}
{"log":"[INFO] [07-10|08:04:04.379] [snapshots:download] Stat                blocks=29500k indices=29500k alloc=2.6GB sys=4.5GB\n","stream":"stderr","time":"2024-07-10T08:04:04.379945122Z"}
{"log":"[INFO] [07-10|08:04:04.380] [2/15 Headers] Waiting for headers...    from=40343081\n","stream":"stderr","time":"2024-07-10T08:04:04.380266205Z"}
{"log":"[INFO] [07-10|08:04:12.627] [2/15 Headers] Processed                 highest=40345477 age=11s headers=2396 in=8.248 blk/sec=290\n","stream":"stderr","time":"2024-07-10T08:04:12.628295403Z"}
{"log":"[INFO] [07-10|08:04:13.895] [5/15 Bodies] Processing bodies...       from=40343081 to=40345477\n","stream":"stderr","time":"2024-07-10T08:04:13.895780229Z"}
{"log":"[INFO] [07-10|08:04:43.897] [5/15 Bodies] Downloading block bodies   block_num=40344374 delivery/sec=3.8MB wasted/sec=486.4KB remaining=1103 delivered=1426 blk/sec=47 cache=10.6MB alloc=3.1GB sys=4.5GB\n","stream":"stderr","time":"2024-07-10T08:04:43.897631672Z"}
{"log":"[INFO] [07-10|08:05:12.915] [5/15 Bodies] Processed                  highest=40345477 blocks=2396 in=59.020 blk/sec=40\n","stream":"stderr","time":"2024-07-10T08:05:12.916086418Z"}
{"log":"[INFO] [07-10|08:05:12.916] [6/15 Senders] Started                   from=40343181 to=40345477\n","stream":"stderr","time":"2024-07-10T08:05:12.916273222Z"}
{"log":"[INFO] [07-10|08:05:14.333] [7/15 Execution] Blocks execution        from=40343181 to=40345477\n","stream":"stderr","time":"2024-07-10T08:05:14.333692067Z"}
{"log":"[WARN] [07-10|08:05:14.345] [7/15 Execution] Execution failed        block=40343182 hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err=\"invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537\"\n","stream":"stderr","time":"2024-07-10T08:05:14.345939903Z"}
{"log":"[INFO] [07-10|08:05:14.349] [7/15 Execution] Completed on            block=40343181\n","stream":"stderr","time":"2024-07-10T08:05:14.349463871Z"}
{"log":"[WARN] [07-10|08:05:14.968] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. \n","stream":"stderr","time":"2024-07-10T08:05:14.969026355Z"}
{"log":"[INFO] [07-10|08:05:14.969] RPC Daemon notified of new headers       from=40343081 to=40343181 amount=100 hash=0xd0420f6083a3aefd6f61ce6d70f4e57ffba85181c0eedbb7058e7e19a745b513 header sending=241.487µs log sending=506ns\n","stream":"stderr","time":"2024-07-10T08:05:14.969335776Z"}
{"log":"[INFO] [07-10|08:05:14.969] Timings (slower than 50ms)               Snapshots=3.156s Headers=8.248s BlockHashes=1.267s Bodies=59.02s Senders=1.417s Unwind Bodies=286ms Unwind Headers=298ms\n","stream":"stderr","time":"2024-07-10T08:05:14.96941726Z"}
{"log":"[INFO] [07-10|08:05:14.969] [2/15 Headers] Waiting for headers...    from=40343181\n","stream":"stderr","time":"2024-07-10T08:05:14.970177964Z"}
{"log":"[WARN] [07-10|08:05:15.207] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:15.207561868Z"}
{"log":"[WARN] [07-10|08:05:15.721] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:15.721900692Z"}
{"log":"[WARN] [07-10|08:05:15.747] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:15.748051268Z"}
{"log":"[WARN] [07-10|08:05:16.149] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.149844194Z"}
{"log":"[WARN] [07-10|08:05:16.204] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.205012046Z"}
{"log":"[WARN] [07-10|08:05:16.226] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.226320617Z"}
{"log":"[WARN] [07-10|08:05:16.323] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.323918672Z"}
{"log":"[WARN] [07-10|08:05:16.351] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.352072844Z"}
{"log":"[WARN] [07-10|08:05:16.430] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.430787239Z"}
{"log":"[WARN] [07-10|08:05:16.437] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:16.437721574Z"}
{"log":"[WARN] [07-10|08:05:17.255] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.255828486Z"}
{"log":"[WARN] [07-10|08:05:17.296] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.296709641Z"}
{"log":"[WARN] [07-10|08:05:17.336] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.336322493Z"}
{"log":"[WARN] [07-10|08:05:17.359] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.359889889Z"}
{"log":"[WARN] [07-10|08:05:17.583] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.583540937Z"}
{"log":"[WARN] [07-10|08:05:17.684] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.684709528Z"}
{"log":"[WARN] [07-10|08:05:17.699] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.699385734Z"}
{"log":"[WARN] [07-10|08:05:17.714] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.714664318Z"}
{"log":"[WARN] [07-10|08:05:17.726] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.726360648Z"}
{"log":"[WARN] [07-10|08:05:17.750] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.750484216Z"}
{"log":"[WARN] [07-10|08:05:17.776] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.77700675Z"}
{"log":"[WARN] [07-10|08:05:17.780] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.780392125Z"}
{"log":"[WARN] [07-10|08:05:17.804] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182\n","stream":"stderr","time":"2024-07-10T08:05:17.80444825Z"}
WangXiaopai595 commented 1 month ago

Is this problem unsolvable?

temuera commented 1 month ago

@blxdyx bsc-erigon-temp_fix not work.

INFO[07-11|13:37:19.737] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|13:37:19.742] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
INFO[07-11|13:37:22.852] [snapshots:all] Stat                     blocks=40253k indices=40253k alloc=3.0GB sys=3.2GB
INFO[07-11|13:37:22.856] [snapshots:all] Stat                     blocks=0k indices=0k alloc=3.0GB sys=3.2GB
INFO[07-11|13:37:22.865] Opening Database                         label=parlia path=/tank/bsc_erigon/data/parlia
mdbx_setup_dxb:16117 opening after an unclean shutdown, but boot-id(83b0c330b86348f7-796b8ca420014046) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|13:37:22.866] Opening Database                         label=blob path=/tank/bsc_erigon/data/blobs/blob
INFO[07-11|13:37:24.082] [10/15 CallTraces] Unwind                from=40343181 to=40343161
INFO[07-11|13:37:24.308] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=true
INFO[07-11|13:37:24.333] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=false
INFO[07-11|13:37:24.402] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=true codes=false
INFO[07-11|13:37:24.563] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=AccountChangeSet
INFO[07-11|13:37:24.603] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=StorageChangeSet
EROR[07-11|13:37:25.404] [9/15 IntermediateHashes] block Number 40343161, block hash 0xadd61ef1d94009f180c11f329ee0379c53e49d862c36439129f1d8820f6a7e2c, err = wrong trie root: 5ad231002c0f58070379de4ab617b7f0d4bb2f745217060f10b7e65b9eae1816, expected (from header): 7ffd370e413891d2e0d6e43d7d3a2aef0cf86772eafab24271b2f7d9c83ce882
WangXiaopai595 commented 1 month ago

@blxdyx bsc-erigon-temp_fix not work.

INFO[07-11|13:37:19.737] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|13:37:19.742] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
INFO[07-11|13:37:22.852] [snapshots:all] Stat                     blocks=40253k indices=40253k alloc=3.0GB sys=3.2GB
INFO[07-11|13:37:22.856] [snapshots:all] Stat                     blocks=0k indices=0k alloc=3.0GB sys=3.2GB
INFO[07-11|13:37:22.865] Opening Database                         label=parlia path=/tank/bsc_erigon/data/parlia
mdbx_setup_dxb:16117 opening after an unclean shutdown, but boot-id(83b0c330b86348f7-796b8ca420014046) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|13:37:22.866] Opening Database                         label=blob path=/tank/bsc_erigon/data/blobs/blob
INFO[07-11|13:37:24.082] [10/15 CallTraces] Unwind                from=40343181 to=40343161
INFO[07-11|13:37:24.308] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=true
INFO[07-11|13:37:24.333] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=false codes=false
INFO[07-11|13:37:24.402] [8/15 HashState] Unwinding started       from=40343181 to=40343161 storage=true codes=false
INFO[07-11|13:37:24.563] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=AccountChangeSet
INFO[07-11|13:37:24.603] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343161 csbucket=StorageChangeSet
EROR[07-11|13:37:25.404] [9/15 IntermediateHashes] block Number 40343161, block hash 0xadd61ef1d94009f180c11f329ee0379c53e49d862c36439129f1d8820f6a7e2c, err = wrong trie root: 5ad231002c0f58070379de4ab617b7f0d4bb2f745217060f10b7e65b9eae1816, expected (from header): 7ffd370e413891d2e0d6e43d7d3a2aef0cf86772eafab24271b2f7d9c83ce882

He seems to need the synchronization log before block 40343161. Do you have it on your end? Mine was lost due to restarting Docker

temuera commented 1 month ago

@WangXiaopai595 啥意思？我有尝试过回退到更早的块，但是同样不行。加个WX啊同学，我的78085 erigon不是在 unwind，就是在 unwind的路上。真费神。

blxdyx commented 1 month ago

Is this problem unsolvable?

I am trying to fix. Still in test.

blxdyx commented 1 month ago

@temuera @WangXiaopai595

Try fix follow those step: use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind.

  1. make integration && make erigon
  2. ./build/bin/integration state_stages --unwind=1 --chain=bsc --datadir=./data
  3. ./build/bin/integration stage_headers --unwind=1 --chain=bsc --datadir=./data
  4. add --stage.step=1 flag to start command and then restart.
  5. If it pass the block 40343181, then can switch to v1.2.12

Give me feed back if it works, thanks

WangXiaopai595 commented 1 month ago

@temuera @WangXiaopai595

Try fix follow those step: use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind.

  1. make integration && make erigon
  2. ./build/bin/integration state_stages --unwind=1 --chain=bsc --datadir=./data
  3. ./build/bin/integration stage_headers --unwind=1 --chain=bsc --datadir=./data
  4. add --stage.step=1 flag to start command and then restart.
  5. If it pass the block 40343181, then can switch to v1.2.12

Give me feed back if it works, thanks

Still failed

integration state_stages --unwind=1 --chain=bsc --datadir=./data 
INFO[07-11|14:49:47.290] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|14:49:47.311] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|14:49:52.519] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.523] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.581] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:15946 filesize mismatch (expect 98529280b/24055p, have 100663296b/24576p)
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|14:49:52.588] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
INFO[07-11|14:50:03.654] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=true
INFO[07-11|14:50:04.096] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=false
INFO[07-11|14:50:04.641] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=true codes=false
INFO[07-11|14:50:05.963] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=AccountChangeSet
INFO[07-11|14:50:06.459] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=StorageChangeSet
EROR[07-11|14:50:12.698] [9/15 IntermediateHashes] block Number 40343180, block hash 0xb4abb99ee61f474ecbf94de57adffb1359871852055797e2394bcb352b52ce94, err = wrong trie root: 8607b05aca7da11632f52ff54220064e43f6d37af8a289feb74735a1940efea7, expected (from header): 5a1525f7a9cdeac03018695fbf4f863a2b4af60179b0c6b0ccf4760abe2f77b0
blxdyx commented 1 month ago

@temuera @WangXiaopai595 Try fix follow those step: use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind.

  1. make integration && make erigon
  2. ./build/bin/integration state_stages --unwind=1 --chain=bsc --datadir=./data
  3. ./build/bin/integration stage_headers --unwind=1 --chain=bsc --datadir=./data
  4. add --stage.step=1 flag to start command and then restart.
  5. If it pass the block 40343181, then can switch to v1.2.12

Give me feed back if it works, thanks

Still failed

integration state_stages --unwind=1 --chain=bsc --datadir=./data 
INFO[07-11|14:49:47.290] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|14:49:47.311] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|14:49:52.519] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.523] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.581] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:15946 filesize mismatch (expect 98529280b/24055p, have 100663296b/24576p)
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|14:49:52.588] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
INFO[07-11|14:50:03.654] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=true
INFO[07-11|14:50:04.096] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=false
INFO[07-11|14:50:04.641] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=true codes=false
INFO[07-11|14:50:05.963] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=AccountChangeSet
INFO[07-11|14:50:06.459] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=StorageChangeSet
EROR[07-11|14:50:12.698] [9/15 IntermediateHashes] block Number 40343180, block hash 0xb4abb99ee61f474ecbf94de57adffb1359871852055797e2394bcb352b52ce94, err = wrong trie root: 8607b05aca7da11632f52ff54220064e43f6d37af8a289feb74735a1940efea7, expected (from header): 5a1525f7a9cdeac03018695fbf4f863a2b4af60179b0c6b0ccf4760abe2f77b0

You logs show you never use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind.

WangXiaopai595 commented 1 month ago

@temuera @WangXiaopai595 尝试按照以下步骤修复：使用此分支：https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind。

  1. 进行集成 && 进行 erigon
  2. ./build/bin/integration state_stages --unwind=1 --chain=bsc --datadir=./data
  3. ./build/bin/integration stage_headers --unwind=1 --chain=bsc --datadir=./data
  4. 添加--stage.step=1标志以启动命令然后重新启动。
  5. 如果通过了 40343181 块，则可以切换到 v1.2.12

如果有效请给我反馈，谢谢

还是失败了

integration state_stages --unwind=1 --chain=bsc --datadir=./data 
INFO[07-11|14:49:47.290] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|14:49:47.311] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|14:49:52.519] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.523] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.3GB sys=2.4GB
INFO[07-11|14:49:52.581] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:15946 filesize mismatch (expect 98529280b/24055p, have 100663296b/24576p)
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|14:49:52.588] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
INFO[07-11|14:50:03.654] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=true
INFO[07-11|14:50:04.096] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=false codes=false
INFO[07-11|14:50:04.641] [8/15 HashState] Unwinding started       from=40343181 to=40343180 storage=true codes=false
INFO[07-11|14:50:05.963] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=AccountChangeSet
INFO[07-11|14:50:06.459] [9/15 IntermediateHashes] Unwinding      from=40343181 to=40343180 csbucket=StorageChangeSet
EROR[07-11|14:50:12.698] [9/15 IntermediateHashes] block Number 40343180, block hash 0xb4abb99ee61f474ecbf94de57adffb1359871852055797e2394bcb352b52ce94, err = wrong trie root: 8607b05aca7da11632f52ff54220064e43f6d37af8a289feb74735a1940efea7, expected (from header): 5a1525f7a9cdeac03018695fbf4f863a2b4af60179b0c6b0ccf4760abe2f77b0

您的日志显示您从未使用过此分支：https://github.com/blxdyx/bsc-erigon/tree/for_test_unwind。

I downloaded the zip file, extracted it, and compiled it

blxdyx commented 1 month ago

@WangXiaopai595 Anyway, you log show you still in this branch: https://github.com/blxdyx/bsc-erigon/tree/temp_fix

WangXiaopai595 commented 1 month ago

Sorry, I unzipped the wrong file. I tried again successfully and the node is already running. I will check later to see if it can synchronize properly

blxdyx commented 1 month ago

Sorry, I unzipped the wrong file. I tried again successfully and the node is already running. I will check later to see if it can synchronize properly

Ok, this branch only for fix, if synchronize properly, just switch to v1.2.12 and remove flag --stage.step=1

saravadeanil commented 1 month ago

Sorry, I unzipped the wrong file. I tried again successfully and the node is already running. I will check later to see if it can synchronize properly

Hi @WangXiaopai595 Can you please share the workable procedure you performed to recover the node?

After you confirm/verify everything is working as expected and node is syncing.

WangXiaopai595 commented 1 month ago

抱歉，我解压了错误的文件。我重试成功，节点已运行。我稍后会检查它是否可以正常同步。

好的，此分支仅用于修复，如果同步正确，只需切换到 v1.2.12 并删除标志 --stage.step=1

It still doesn't work

[INFO] [07-11|10:49:23.823] logging to file system                   log dir=./log file prefix=erigon log level=info json=false
[INFO] [07-11|10:49:23.824] Enabling metrics export to prometheus    path=http://0.0.0.0:6060/debug/metrics/prometheus
[INFO] [07-11|10:49:23.824] Build info                               git_branch= git_tag= git_commit=
[INFO] [07-11|10:49:23.824] Starting Erigon on BSC mainnet... 
[INFO] [07-11|10:49:23.826] Maximum peer count                       ETH=1000 total=1000
[INFO] [07-11|10:49:23.826] starting HTTP APIs                       port=8545 APIs=eth,erigon,engine
[INFO] [07-11|10:49:23.827] torrent verbosity                        level=WRN
[INFO] [07-11|10:49:24.102] Set global gas cap                       cap=50000000
[INFO] [07-11|10:49:24.114] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [07-11|10:49:24.114] Opening Database                         label=chaindata path=/home/erigon/data/chaindata
[INFO] [07-11|10:49:24.118] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
[INFO] [07-11|10:49:24.137] Initialised chain configuration          config="{ChainID: 56 Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024, Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime 1705996800, FeynmanTime 1713419340, FeynmanFixTime 1713419340, CancunTime 1718863500, HaberTime 1718863500, HaberFixTime <nil>, Engine: parlia}" genesis=0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b
[INFO] [07-11|10:49:52.064] Initialising Ethereum protocol           network=56
[INFO] [07-11|10:49:52.064] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:15946 filesize mismatch (expect 100663296b/24576p, have 98529280b/24055p)
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
[INFO] [07-11|10:49:52.072] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
[INFO] [07-11|10:49:53.317] Starting private RPC server              on=localhost:9090
[INFO] [07-11|10:49:53.317] new subscription to logs established 
[INFO] [07-11|10:49:53.318] rpc filters: subscribing to Erigon events 
[INFO] [07-11|10:49:53.318] new subscription to newHeaders established 
[INFO] [07-11|10:49:53.320] Reading JWT secret                       path=/home/erigon/data/jwt.hex
[INFO] [07-11|10:49:53.320] JsonRpc endpoint opened                  ws=false ws.compression=true grpc=false http.url=127.0.0.1:8545
[INFO] [07-11|10:49:53.320] HTTP endpoint opened for Engine API      url=127.0.0.1:8551 ws=true ws.compression=true
[WARN] [07-11|10:49:53.337] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[WARN] [07-11|10:49:53.337] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[WARN] [07-11|10:49:53.337] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[WARN] [07-11|10:49:53.337] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[INFO] [07-11|10:49:53.346] Started P2P networking                   version=68 self=enode://49e8672b6c94abf8c18292ade56dcacdad66b4ccf99a8c7d90e861a691adb90db51372308e24a23cfb46a9b0977d54429aa1644dc9b5383b3658cca39e9d05fe@127.0.0.1:30311 name=erigon/v1.2.12/linux-amd64/go1.21.6
[INFO] [07-11|10:49:53.353] Mapped network port                      proto=tcp extport=30311 intport=30311 interface=NAT-PMP(192.168.1.1)
[INFO] [07-11|10:49:53.354] Mapped network port                      proto=udp extport=30311 intport=30311 interface=NAT-PMP(192.168.1.1)
[INFO] [07-11|10:49:53.376] [1/15 Snapshots] Requesting downloads 
[WARN] [07-11|10:49:57.373] [1/15 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=30499999 min_in_db=31753500
[INFO] [07-11|10:49:57.374] [snapshots:download] Stat                blocks=30500k indices=30500k alloc=2.7GB sys=4.9GB
[INFO] [07-11|10:49:57.374] [2/15 Headers] Waiting for headers...    from=40343161
[INFO] [07-11|10:50:27.376] [2/15 Headers] Inserting headers         progress=40353305 queue=1
[INFO] [07-11|10:50:57.377] [2/15 Headers] Inserting headers         progress=40363873 queue=1
[INFO] [07-11|10:51:08.453] [2/15 Headers] Processed                 highest=40367941 age=2s headers=24780 in=71.079 blk/sec=348
[INFO] [07-11|10:51:08.453] [2/15 Headers] DONE                      in=1m11.079064032s
[INFO] [07-11|10:51:12.225] [5/15 Bodies] Processing bodies...       from=40343161 to=40367941
[INFO] [07-11|10:51:42.227] [5/15 Bodies] Downloading block bodies   block_num=40344437 delivery/sec=3.4MB wasted/sec=511.2KB remaining=23542 delivered=1278 blk/sec=42 cache=75.2KB alloc=2.9GB sys=5.0GB
[INFO] [07-11|10:52:12.226] [5/15 Bodies] Downloading block bodies   block_num=40346104 delivery/sec=4.9MB wasted/sec=936.1KB remaining=21836 delivered=3102 blk/sec=103 cache=13.0MB alloc=4.6GB sys=5.2GB
[INFO] [07-11|10:52:42.228] [5/15 Bodies] Writing block bodies       block_num=40348077 remaining=19864 alloc=4.0GB sys=5.3GB
[INFO] [07-11|10:52:52.065] [p2p] GoodPeers                          eth68=31
[INFO] [07-11|10:52:52.300] [mem] memory stats                       Rss=6.7GB Size=15.5TB Pss=6.7GB SharedClean=4.0KB SharedDirty=0B PrivateClean=1.1GB PrivateDirty=5.6GB Referenced=6.7GB Anonymous=5.6GB Swap=0B alloc=4.4GB sys=5.3GB
[INFO] [07-11|10:53:12.228] [5/15 Bodies] Downloading block bodies   block_num=40349349 delivery/sec=8.2MB wasted/sec=803.8KB remaining=18591 delivered=6317 blk/sec=210 cache=10.6MB alloc=2.9GB sys=5.3GB
[INFO] [07-11|10:53:42.227] [5/15 Bodies] Downloading block bodies   block_num=40350868 delivery/sec=4.7MB wasted/sec=361.3KB remaining=17072 delivered=7826 blk/sec=260 cache=11.1MB alloc=4.4GB sys=5.3GB
[INFO] [07-11|10:54:12.226] [5/15 Bodies] Downloading block bodies   block_num=40352089 delivery/sec=3.3MB wasted/sec=780.2KB remaining=15880 delivered=9011 blk/sec=300 cache=6.8MB alloc=3.3GB sys=5.3GB
[INFO] [07-11|10:54:42.226] [5/15 Bodies] Downloading block bodies   block_num=40353443 delivery/sec=3.6MB wasted/sec=296.0KB remaining=14497 delivered=10347 blk/sec=344 cache=6.3MB alloc=4.6GB sys=5.3GB
[INFO] [07-11|10:55:12.227] [5/15 Bodies] Downloading block bodies   block_num=40354519 delivery/sec=3.5MB wasted/sec=1.1MB remaining=13421 delivered=11578 blk/sec=385 cache=19.3MB alloc=3.5GB sys=5.3GB
[INFO] [07-11|10:55:42.226] [5/15 Bodies] Downloading block bodies   block_num=40356592 delivery/sec=4.7MB wasted/sec=1.1MB remaining=11348 delivered=13489 blk/sec=449 cache=3.7MB alloc=3.0GB sys=5.4GB
[INFO] [07-11|10:55:52.065] [p2p] GoodPeers                          eth68=53
[INFO] [07-11|10:55:52.340] [mem] memory stats                       Rss=7.3GB Size=15.5TB Pss=7.3GB SharedClean=4.0KB SharedDirty=0B PrivateClean=1.3GB PrivateDirty=6.0GB Referenced=7.3GB Anonymous=5.9GB Swap=0B alloc=3.4GB sys=5.4GB
[INFO] [07-11|10:56:12.227] [5/15 Bodies] Downloading block bodies   block_num=40358985 delivery/sec=4.3MB wasted/sec=431.6KB remaining=8955 delivered=16027 blk/sec=534 cache=8.1MB alloc=4.5GB sys=5.4GB
[INFO] [07-11|10:56:42.226] [5/15 Bodies] Downloading block bodies   block_num=40361381 delivery/sec=4.3MB wasted/sec=583.0KB remaining=6559 delivered=18487 blk/sec=616 cache=17.3MB alloc=3.5GB sys=5.4GB
[INFO] [07-11|10:57:12.226] [5/15 Bodies] Downloading block bodies   block_num=40364388 delivery/sec=4.2MB wasted/sec=743.3KB remaining=3764 delivered=21241 blk/sec=708 cache=805.3KB alloc=2.7GB sys=5.4GB
[INFO] [07-11|10:57:42.229] [5/15 Bodies] Downloading block bodies   block_num=40365930 delivery/sec=3.4MB wasted/sec=1.1MB remaining=2010 delivered=22933 blk/sec=764 cache=10.5MB alloc=4.0GB sys=5.4GB
[INFO] [07-11|10:58:12.227] [5/15 Bodies] Downloading block bodies   block_num=40367311 delivery/sec=3.2MB wasted/sec=970.2KB remaining=629 delivered=24217 blk/sec=807 cache=6.5MB alloc=3.0GB sys=5.4GB
[INFO] [07-11|10:58:34.259] [5/15 Bodies] Processed                  highest=40367941 blocks=24780 in=442.034 blk/sec=56
[INFO] [07-11|10:58:34.259] [5/15 Bodies] DONE                       in=7m22.034641878s
[INFO] [07-11|10:58:34.259] [6/15 Senders] Started                   from=40343181 to=40367941
[INFO] [07-11|10:58:46.376] [7/15 Execution] Blocks execution        from=40343181 to=40367941
[WARN] [07-11|10:58:46.380] [7/15 Execution] Execution failed        block=40343182 hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 err="invalid block: could not apply tx 2 from block 40343182 [0x2d24f8caa239758d699ce275c31cbddaaeabf3fca8123c4e9015f1304f820c33]: nonce too low: address 0x595Db43fb04e1422222ecB327d98dDafDC814241, tx: 1192536 state: 1192537"
[INFO] [07-11|10:58:46.381] [7/15 Execution] Completed on            block=40343181
[WARN] [07-11|10:58:48.924] sentry.StatusDataProvider: The canonical chain current header not found in the database. Check the database consistency. Using genesis as a fallback. 
[INFO] [07-11|10:58:48.924] RPC Daemon notified of new headers       from=40343161 to=40343181 amount=20 hash=0xd0420f6083a3aefd6f61ce6d70f4e57ffba85181c0eedbb7058e7e19a745b513 header sending=71.51µs log sending=502ns
[INFO] [07-11|10:58:48.927] Timings (slower than 50ms)               Snapshots=4.027s Headers=1m11.079s BlockHashes=3.771s Bodies=7m22.034s Senders=12.116s Unwind Bodies=1.295s Unwind Headers=1.244s
[INFO] [07-11|10:58:48.928] [2/15 Headers] Waiting for headers...    from=40343181
[WARN] [07-11|10:58:50.151] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:50.158] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:50.209] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:50.227] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.150] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.171] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.193] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.207] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.220] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:51.369] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[INFO] [07-11|10:58:52.064] [p2p] GoodPeers                          eth68=77
[WARN] [07-11|10:58:52.148] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:52.170] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:52.186] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[WARN] [07-11|10:58:52.230] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
[INFO] [07-11|10:58:52.381] [mem] memory stats                       Rss=8.7GB Size=15.5TB Pss=8.7GB SharedClean=4.0KB SharedDirty=0B PrivateClean=2.3GB PrivateDirty=6.4GB Referenced=8.7GB Anonymous=6.4GB Swap=0B alloc=4.1GB sys=5.6GB
[WARN] [07-11|10:58:52.585] [downloader] Rejected header marked as bad hash=0x2545857dcb180ccbe191e50b27e79f3c07797b70fb3d94b70211253cf6acb0f3 height=40343182
blxdyx commented 1 month ago

show print_stages

[INFO] [07-11|10:49:57.374] [2/15 Headers] Waiting for headers... from=40343161 this logs show you unwind to 40343161, you should unwind to 40343180. You have follow my step?

WangXiaopai595 commented 1 month ago

print_stages --chain=bsc --datadir=./data

./bsc-erigon-for_test_unwind/build/bin/integration state_stages print_stages --chain=bsc --datadir=./data
INFO[07-11|15:28:28.643] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|15:28:28.653] [db] open                                label=chaindata sizeLimit=10TB pageSize=16384
INFO[07-11|15:28:33.127] [snapshots:all] Stat                     blocks=30500k indices=30500k alloc=2.4GB sys=2.5GB
INFO[07-11|15:28:33.130] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.4GB sys=2.5GB
INFO[07-11|15:28:33.167] Opening Database                         label=parlia path=/home/erigon/data/parlia
mdbx_setup_dxb:15946 filesize mismatch (expect 98590720b/24070p, have 100663296b/24576p)
mdbx_setup_dxb:16114 opening after an unclean shutdown, but boot-id(bf9c1e97497ae15e-3656c57cc4894668) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[07-11|15:28:33.175] Opening Database                         label=blob path=/home/erigon/data/blobs/blob
temuera commented 1 month ago

Print stages  1, unwind to 40341143 success. 2,use bsc-erigon-for_test_unwind/erigon with --stage.step=1 

WARN[07-11|15:29:00.638] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.652] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.670] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.678] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
./integration print_stages --chain=bsc --datadir=./data
INFO[07-11|15:31:17.945] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|15:31:17.948] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
INFO[07-11|15:31:22.008] [snapshots:all] Stat                     blocks=40253k indices=40253k alloc=3.0GB sys=3.4GB
INFO[07-11|15:31:22.012] [snapshots:all] Stat                     blocks=0k indices=0k alloc=3.0GB sys=3.4GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

                                 stage_at        prune_at
Snapshots                        40343180        0
Headers                          40341142        0
BorHeimdall                      0               0
BlockHashes                      40341142        0
Bodies                           40341142        0
Senders                          40341142        0
Execution                        40341142        40341142
Translation                      0               0
HashState                        40341142        0
IntermediateHashes               40341142        40341142
AccountHistoryIndex              40341142        0
StorageHistoryIndex              40341142        0
LogIndex                         40341142        0
CallTraces                       40341142        40341142
TxLookup                         40341142        40253000
Finish                           40341142        0
--
prune distance: 

blocks.v2: true, blocks=40252999, segments=40252999, indices=40252999
blocks.bor.v2: segments=0, indices=0

history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,0)

sequence: EthTx=6012387345, NonCanonicalTx=38520

in db: first header 40253000, last header 40341143, first body 40253000, last body 40341143
--
blxdyx commented 1 month ago

Print stages  1, unwind to 40341143 success. 2,use bsc-erigon-for_test_unwind/erigon with --stage.step=1

WARN[07-11|15:29:00.638] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.652] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.670] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
WARN[07-11|15:29:00.678] [downloader] Rejected header marked as bad hash=0x2393dc067c9cdf1e88b3923a3fe81808e97cc07f8b5e718eb706db37ad45af40 height=40341143
./integration print_stages --chain=bsc --datadir=./data
INFO[07-11|15:31:17.945] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[07-11|15:31:17.948] [db] open                                label=chaindata sizeLimit=12TB pageSize=16384
INFO[07-11|15:31:22.008] [snapshots:all] Stat                     blocks=40253k indices=40253k alloc=3.0GB sys=3.4GB
INFO[07-11|15:31:22.012] [snapshots:all] Stat                     blocks=0k indices=0k alloc=3.0GB sys=3.4GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

                                 stage_at        prune_at
Snapshots                        40343180        0
Headers                          40341142        0
BorHeimdall                      0               0
BlockHashes                      40341142        0
Bodies                           40341142        0
Senders                          40341142        0
Execution                        40341142        40341142
Translation                      0               0
HashState                        40341142        0
IntermediateHashes               40341142        40341142
AccountHistoryIndex              40341142        0
StorageHistoryIndex              40341142        0
LogIndex                         40341142        0
CallTraces                       40341142        40341142
TxLookup                         40341142        40253000
Finish                           40341142        0
--
prune distance: 

blocks.v2: true, blocks=40252999, segments=40252999, indices=40252999
blocks.bor.v2: segments=0, indices=0

history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,0)

sequence: EthTx=6012387345, NonCanonicalTx=38520

in db: first header 40253000, last header 40341143, first body 40253000, last body 40341143
--

Only need unwind 1 block....

blxdyx commented 1 month ago

WARN] [07-11|10:58:51.369]

WARN] [07-11|10:58:51.369]
@WangXiaopai595 Should INFO[07-11|15:28:33.130] Wrong log time? Make sure you have show the right log?

WangXiaopai595 commented 1 month ago

警告] [07-11|10：58：51.369]

警告] [07-11|10：58：51.369] @WangXiaopai595 日志时间错误？请确保您显示了正确的日志？

[WARN] [07-11|15:14:57.008] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.078] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.457] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.465] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.522] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.547] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.567] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.608] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.712] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.737] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.748] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.760] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.787] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.804] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.818] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.844] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.867] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.922] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.929] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.971] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.040] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.049] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.317] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.666] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.682] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.723] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.858] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.875] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.881] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.924] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.928] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
blxdyx commented 1 month ago

警告] [07-11|10：58：51.369]

警告] [07-11|10：58：51.369] @WangXiaopai595 日志时间错误？请确保您显示了正确的日志？

[WARN] [07-11|15:14:57.008] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.078] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.457] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.465] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.522] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.547] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.567] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.608] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.712] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.737] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.748] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.760] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.787] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.804] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:57.818] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.844] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.867] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.922] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.929] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:57.971] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.040] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.049] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.317] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.666] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.682] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.723] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.858] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.875] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.881] [downloader] Rejected header marked as bad hash=0xd6d6c61814c47b3760061e6ee6d5df370f565033142a3bd4d0e76211ef44a26d height=40372820
[WARN] [07-11|15:14:58.924] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187
[WARN] [07-11|15:14:58.928] [downloader] Rejected header marked as bad hash=0x028289d9840359c050516ae079b32b0ff8e08b7724ee3c3530d9d77192e74d08 height=40343187

Send me the log after you restart. Make sure you have make erigon as the step?

blxdyx commented 1 month ago

40341143

@temuera Don't do any thing. I will cherry-pick a branch just for you.

blxdyx commented 1 month ago

@temuera use this branch: https://github.com/blxdyx/bsc-erigon/tree/for_temuera make erigon and restart with --stage.step=1

insider89 commented 1 month ago

@blxdyx do you plan to provide docker image for that fix? We are running erigon on k8s cluster

blxdyx commented 1 month ago

@blxdyx do you plan to provide docker image for that fix? We are running erigon on k8s cluster

Yes, once i make sure it's no problem.

temuera commented 1 month ago

@blxdyx

testing.....

temuera commented 1 month ago

@blxdyx
syncing...... let's see if we can get past 40343181 :(

blxdyx commented 1 month ago

40343181

@temuera How about it now?

temuera commented 1 month ago

40343181

@temuera How about it now?

syncing.... need more time.

INFO[07-11|16:06:08.256] RPC Daemon notified of new headers       from=40341350 to=40341351 amount=1 hash=0x25969163b0de53aa74d47f442312b329cd03e459310f6fa93edecd1831fa1e9f header sending=33.363µs log sending=331ns
INFO[07-11|16:06:08.256] Timings (slower than 50ms)               Bodies=647ms Execution=207ms HashState=153ms IntermediateHashes=320ms CallTraces=69ms AccountHistoryIndex=64ms StorageHistoryIndex=119ms LogIndex=72ms
INFO[07-11|16:06:08.256] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=176.0MB ReclaimableSpace=704.1GB
INFO[07-11|16:06:08.391] [2/15 Headers] Waiting for headers...    from=40341351
INFO[07-11|16:06:08.393] [2/15 Headers] Processed                 highest=40341352 age=1d3h31m headers=1 in=0.002 blk/sec=513
INFO[07-11|16:06:10.554] [7/15 Execution] Completed on            block=40341352
INFO[07-11|16:06:11.199] RPC Daemon notified of new headers       from=40341351 to=40341352 amount=1 hash=0x3563cad9f1287dc42f29ebf8703e4895f15cae7b973fd00de6f9865b3dc135bc header sending=34.405µs log sending=541ns
INFO[07-11|16:06:11.199] Timings (slower than 50ms)               Bodies=2.004s Execution=146ms HashState=96ms IntermediateHashes=216ms StorageHistoryIndex=68ms
INFO[07-11|16:06:11.199] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=176.0MB ReclaimableSpace=704.1GB
INFO[07-11|16:06:11.292] [2/15 Headers] Waiting for headers...    from=40341352
INFO[07-11|16:06:11.294] [2/15 Headers] Processed                 highest=40341353 age=1d3h31m headers=1 in=0.002
blxdyx commented 1 month ago

40343181

@temuera How about it now?

syncing.... need more time.

INFO[07-11|16:06:08.256] RPC Daemon notified of new headers       from=40341350 to=40341351 amount=1 hash=0x25969163b0de53aa74d47f442312b329cd03e459310f6fa93edecd1831fa1e9f header sending=33.363µs log sending=331ns
INFO[07-11|16:06:08.256] Timings (slower than 50ms)               Bodies=647ms Execution=207ms HashState=153ms IntermediateHashes=320ms CallTraces=69ms AccountHistoryIndex=64ms StorageHistoryIndex=119ms LogIndex=72ms
INFO[07-11|16:06:08.256] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=176.0MB ReclaimableSpace=704.1GB
INFO[07-11|16:06:08.391] [2/15 Headers] Waiting for headers...    from=40341351
INFO[07-11|16:06:08.393] [2/15 Headers] Processed                 highest=40341352 age=1d3h31m headers=1 in=0.002 blk/sec=513
INFO[07-11|16:06:10.554] [7/15 Execution] Completed on            block=40341352
INFO[07-11|16:06:11.199] RPC Daemon notified of new headers       from=40341351 to=40341352 amount=1 hash=0x3563cad9f1287dc42f29ebf8703e4895f15cae7b973fd00de6f9865b3dc135bc header sending=34.405µs log sending=541ns
INFO[07-11|16:06:11.199] Timings (slower than 50ms)               Bodies=2.004s Execution=146ms HashState=96ms IntermediateHashes=216ms StorageHistoryIndex=68ms
INFO[07-11|16:06:11.199] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=176.0MB ReclaimableSpace=704.1GB
INFO[07-11|16:06:11.292] [2/15 Headers] Waiting for headers...    from=40341352
INFO[07-11|16:06:11.294] [2/15 Headers] Processed                 highest=40341353 age=1d3h31m headers=1 in=0.002

@temuera any update?

temuera commented 1 month ago

@blxdyx 

NFO[07-11|16:24:30.348] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=175.9MB ReclaimableSpace=703.6GB
INFO[07-11|16:24:30.431] [2/15 Headers] Waiting for headers...    from=40341933
INFO[07-11|16:24:30.433] [2/15 Headers] Processed                 highest=40341934 age=1d3h20m headers=1 in=0.002 blk/sec=499
INFO[07-11|16:24:31.230] [7/15 Execution] Completed on            block=40341934
INFO[07-11|16:24:31.885] RPC Daemon notified of new headers       from=40341933 to=40341934 amount=1 hash=0x20606de94da0212e8a9636e67f0d98f887c68c144b57780fb5b4be93313b8f11 header sending=25.909µs log sending=511ns
INFO[07-11|16:24:31.885] Timings (slower than 50ms)               Bodies=658ms Execution=125ms HashState=116ms IntermediateHashes=203ms CallTraces=53ms StorageHistoryIndex=71ms
INFO[07-11|16:24:31.885] Tables                                   PlainState=315.2GB AccountChangeSet=733.7GB StorageChangeSet=1.6TB BlockTransaction=4.0GB TransactionLog=3.2TB FreeList=175.9MB ReclaimableSpace=703.6GB
INFO[07-11|16:24:31.944] [2/15 Headers] Waiting for headers...    from=40341934
INFO[07-11|16:24:31.946] [2/15 Headers] Processed                 highest=40341935 age=1d3h20m headers=1 in=0.002 blk/sec=497
INFO[07-11|16:24:33.394] [7/15 Execution] Completed on            block=40341935
WangXiaopai595 commented 1 month ago

It seems to be running normally now

[INFO] [07-11|16:44:43.232] [7/15 Execution] Blocks execution        from=40343183 to=40374775
[INFO] [07-11|16:44:59.474] [p2p] GoodPeers                          eth68=74
[INFO] [07-11|16:45:00.097] [mem] memory stats                       Rss=9.4GB Size=15.5TB Pss=9.4GB SharedClean=4.0KB SharedDirty=0B PrivateClean=3.0GB PrivateDirty=6.4GB Referenced=9.4GB Anonymous=6.4GB Swap=0B alloc=2.8GB sys=5.8GB
^[[A[INFO] [07-11|16:45:13.421] [7/15 Execution] Executed blocks         number=40343289 blk/s=3.5 tx/s=431.0 Mgas/s=49.9 gasState=0.00 batch=2.5MB alloc=3.2GB sys=5.8GB
[INFO] [07-11|16:45:43.471] [7/15 Execution] Executed blocks         number=40343408 blk/s=4.0 tx/s=482.5 Mgas/s=53.0 gasState=0.01 batch=4.7MB alloc=4.1GB sys=5.8GB
[INFO] [07-11|16:46:13.447] [7/15 Execution] Executed blocks         number=40343536 blk/s=4.3 tx/s=514.7 Mgas/s=57.8 gasState=0.01 batch=6.8MB alloc=2.7GB sys=5.8GB
[INFO] [07-11|16:46:43.309] [7/15 Execution] Executed blocks         number=40343678 blk/s=4.8 tx/s=512.3 Mgas/s=58.7 gasState=0.01 batch=8.9MB alloc=3.6GB sys=5.8GB
[INFO] [07-11|16:47:13.268] [7/15 Execution] Executed blocks         number=40343832 blk/s=5.1 tx/s=562.8 Mgas/s=62.1 gasState=0.02 batch=11.2MB alloc=4.7GB sys=5.8GB
[INFO] [07-11|16:47:43.715] [7/15 Execution] Executed blocks         number=40343977 blk/s=4.8 tx/s=547.3 Mgas/s=61.8 gasState=0.02 batch=13.7MB alloc=3.4GB sys=5.8GB
[INFO] [07-11|16:47:59.474] [p2p] GoodPeers                          eth68=85
[INFO] [07-11|16:48:00.737] [mem] memory stats                       Rss=12.4GB Size=15.5TB Pss=12.4GB SharedClean=4.0KB SharedDirty=0B PrivateClean=6.0GB PrivateDirty=6.4GB Referenced=12.4GB Anonymous=6.4GB Swap=0B alloc=4.0GB sys=5.8GB
[INFO] [07-11|16:48:13.573] [7/15 Execution] Executed blocks         number=40344119 blk/s=4.8 tx/s=543.9 Mgas/s=63.2 gasState=0.02 batch=16.1MB alloc=4.4GB sys=5.8GB
[INFO] [07-11|16:48:43.902] [7/15 Execution] Executed blocks         number=40344244 blk/s=4.1 tx/s=545.3 Mgas/s=68.8 gasState=0.03 batch=18.7MB alloc=3.1GB sys=5.8GB
[INFO] [07-11|16:49:13.574] [7/15 Execution] Executed blocks         number=40344378 blk/s=4.5 tx/s=580.6 Mgas/s=70.8 gasState=0.03 batch=21.2MB alloc=4.2GB sys=5.8GB
[INFO] [07-11|16:49:43.430] [7/15 Execution] Executed blocks         number=40344523 blk/s=4.9 tx/s=571.4 Mgas/s=68.6 gasState=0.03 batch=23.5MB alloc=2.9GB sys=5.8GB
temuera commented 1 month ago

@blxdyx passed. it works fine

blxdyx commented 1 month ago

@blxdyx passed. it works fine

Great, i just have put the fix step to the top

insider89 commented 1 month ago

@blxdyx when do you expect to release the docker image?

blxdyx commented 1 month ago

@blxdyx when do you expect to release the docker image?

@insider89 Anyway you need make integration and unwind 1 block. Just replace binary will not work. You need follow the step. If you really need a docker image for it. I will try to build one.

insider89 commented 1 month ago

@blxdyx As I understand this unwind step and --stage.step=1 will work only on the integration command from fix branch, so if I do it on the 1.2.12 docker image it won't work for me, or my understanding is wrong?

blxdyx commented 1 month ago

@blxdyx As I understand this unwind step and --stage.step=1 will work only on the integration command from fix branch, so if I do it on the 1.2.12 docker image it won't work for me, or my understanding is wrong?

Yes, the docker is ready. https://github.com/node-real/bsc-erigon/pkgs/container/bsc-erigon

insider89 commented 1 month ago

Thank you. I am trying it. Could you please confirm that last step should also be running with that docker image?

  1. add --stage.step=1 flag to start command and then restart.

So I should not use 1.2.12 even after unwind finish?

blxdyx commented 1 month ago

Thank you. I am trying it. Could you please confirm that last step should also be running with that docker image? 4. add --stage.step=1 flag to start command and then restart.

So I should not use 1.2.12 even after unwind finish?

Yes, use the docker until you back to the tip. And then use the v1.2.12 or just latest main both ok.

insider89 commented 1 month ago

Thank you. How long it usually takes to unwind. Yesterday I started it with --unwind=20, and it takes 16 hours and was on 14/15 stages. And I've restarted pod:( Is it ok to take so long?

blxdyx commented 1 month ago

Thank you. How long it usually takes to unwind. Yesterday I started it with --unwind=20, and it takes 16 hours and was on 14/15 stages. And I've restarted pod:( Is it ok to take so long?

only need 1-2 second. Make sure you node are stuck at 40343181.

insider89 commented 1 month ago

For me it stuck on different block: 40343186. Does this unwind looks wrong to you? Or it ok, and I should proceed with headers unwind?

Screenshot 2024-07-11 at 15 49 05
blxdyx commented 1 month ago

For me it stuck on different block: 40343186. Does this unwind looks wrong to you? Or it ok, and I should proceed with headers unwind? Screenshot 2024-07-11 at 15 49 05

Show print stages.