Closed du5 closed 2 months ago
btw, In v1.4.5 version, prune can work properly
@du5 Which snapshot do you use to prune?
@du5 Which snapshot do you use to prune?
--history.transactions=0 --syncmode=full --db.engine=pebble --state.scheme=hash
@du5 Which snapshot do you use to prune?
--history.transactions=0 --syncmode=full --db.engine=pebble --state.scheme=hash
Can you give me a snapshot that you'are using to prune block? Do you use muti-database to store block data?
@du5 I want to ensure the step you are using:
geth snapshot prune-block
and geth snapshot prune-state
to prune a complete hash data.Is this right?
link #2436
Hi, @du5 The cause of the problem is: In the multi-database PR, the introduction of "uses the finalized block tag as the indicator for chain freezing" in the chainDb resulted in the retention of only a few dozen blocks. After executing prune-block, a restart typically triggers a "rewound" action ("Head state missing, repairing"), rewinding to blocks that do not exist in the chainDb.Since the block data does not exist at this time, it is not possible to perform block tracking. for example: t1: block height 38486723, chaindb blocks [38486700- 38486723], you may rewound to 38482500, but this block was deleted;
The issue can be resolved by running the command:
./geth_linux snapshot prune-block --datadir PATH --datadir.ancient PATH/chaindata/ancient/chain --block-amount-reserved 90000
This command allows specifying the block-amount-reserved parameter to reserve a sufficient number of blocks.
We will work on optimizing the subsequent code to address this issue as soon as possible.
Hi, @du5 The cause of the problem is: In the multi-database PR, the introduction of "uses the finalized block tag as the indicator for chain freezing" in the chainDb resulted in the retention of only a few dozen blocks. After executing prune-block, a restart typically triggers a "rewound" action ("Head state missing, repairing"), rewinding to blocks that do not exist in the chainDb.Since the block data does not exist at this time, it is not possible to perform block tracking. for example: t1: block height 38486723, chaindb blocks [38486700- 38486723], you may rewound to 38482500, but this block was deleted;
The issue can be resolved by running the command:
./geth_linux snapshot prune-block --datadir PATH --datadir.ancient PATH/chaindata/ancient/chain --block-amount-reserved 90000
This command allows specifying the block-amount-reserved parameter to reserve a sufficient number of blocks.We will work on optimizing the subsequent code to address this issue as soon as possible.
thanks for your help
Hi, @du5 The cause of the problem is: In the multi-database PR, the introduction of "uses the finalized block tag as the indicator for chain freezing" in the chainDb resulted in the retention of only a few dozen blocks. After executing prune-block, a restart typically triggers a "rewound" action ("Head state missing, repairing"), rewinding to blocks that do not exist in the chainDb.Since the block data does not exist at this time, it is not possible to perform block tracking. for example: t1: block height 38486723, chaindb blocks [38486700- 38486723], you may rewound to 38482500, but this block was deleted;
The issue can be resolved by running the command:
./geth_linux snapshot prune-block --datadir PATH --datadir.ancient PATH/chaindata/ancient/chain --block-amount-reserved 90000
This command allows specifying the block-amount-reserved parameter to reserve a sufficient number of blocks.We will work on optimizing the subsequent code to address this issue as soon as possible.
prune
➜ geth snapshot prune-block --datadir PATH --datadir.ancient PATH/geth/chaindata/ancient/chain --block-amount-reserved 90000
INFO [05-15|02:36:36.425] Maximum peer count ETH=50 total=50
INFO [05-15|02:36:36.429] Set global gas cap cap=50,000,000
INFO [05-15|02:36:36.429] Initializing the KZG library backend=gokzg
INFO [05-15|02:36:36.442] Using pebble as the backing database
INFO [05-15|02:36:36.442] Allocated cache and file handles database=/opt/geth.fast2/geth/chaindata cache=409.00MiB handles=524,288 "memory table"=102.25MiB
INFO [05-15|02:36:37.771] Found legacy ancient chain path location=/opt/geth.fast2/geth/chaindata/ancient/chain
INFO [05-15|02:36:37.808] Opened ancient database database=/opt/geth.fast2/geth/chaindata/ancient/chain readonly=false frozen=38,721,904
INFO [05-15|02:36:38.488] file lock existed, waiting for prune recovery and continue err=<nil>
INFO [05-15|02:36:38.488] RecoverInterruption for block prune
INFO [05-15|02:36:38.488] New ancientDB_backup did not exist in interruption scenario
INFO [05-15|02:36:38.489] Using pebble as the backing database
INFO [05-15|02:36:38.489] Allocated cache and file handles database=/opt/geth.fast2/geth/chaindata cache=512.00MiB handles=524,288 "memory table"=128.00MiB
INFO [05-15|02:36:38.517] Found legacy ancient chain path location=/opt/geth.fast2/geth/chaindata/ancient/chain
INFO [05-15|02:36:38.517] Opened ancient database database=/opt/geth.fast2/geth/chaindata/ancient/chain readonly=false frozen=38,721,904
INFO [05-15|02:36:38.518] chainDB opened successfully
INFO [05-15|02:36:38.518] the number of items in ancientDB is itemsOfAncient=239,510
INFO [05-15|02:36:38.518] the oldOffSet is oldOffSet=38,482,394
INFO [05-15|02:36:38.518] new offset/new startBlockNumber is "new offset"=38,631,904
INFO [05-15|02:36:38.520] Opened ancient database database=/opt/geth.fast2/geth/chaindata/ancient/chain_back readonly=false frozen=38,631,904
INFO [05-15|02:36:38.520] prune info "old offset"=38,482,394 "number of items in ancientDB"=239,510 "amount to reserve"=90000
INFO [05-15|02:36:38.520] new offset/new startBlockNumber recorded successfully "new offset"=38,631,904
INFO [05-15|02:36:43.521] block backup process running successfully "current blockNumber for backup"=38,634,259
INFO [05-15|02:36:48.521] block backup process running successfully "current blockNumber for backup"=38,637,734
INFO [05-15|02:36:53.522] block backup process running successfully "current blockNumber for backup"=38,642,135
INFO [05-15|02:36:58.523] block backup process running successfully "current blockNumber for backup"=38,645,184
INFO [05-15|02:37:03.524] block backup process running successfully "current blockNumber for backup"=38,647,775
INFO [05-15|02:37:08.524] block backup process running successfully "current blockNumber for backup"=38,650,309
INFO [05-15|02:37:13.527] block backup process running successfully "current blockNumber for backup"=38,652,635
^[INFO [05-15|02:37:18.528] block backup process running successfully "current blockNumber for backup"=38,654,886
INFO [05-15|02:37:23.531] block backup process running successfully "current blockNumber for backup"=38,657,131
INFO [05-15|02:37:28.533] block backup process running successfully "current blockNumber for backup"=38,659,208
INFO [05-15|02:37:33.534] block backup process running successfully "current blockNumber for backup"=38,661,422
INFO [05-15|02:37:38.535] block backup process running successfully "current blockNumber for backup"=38,663,895
INFO [05-15|02:37:43.535] block backup process running successfully "current blockNumber for backup"=38,667,098
INFO [05-15|02:37:48.536] block backup process running successfully "current blockNumber for backup"=38,670,661
INFO [05-15|02:37:53.536] block backup process running successfully "current blockNumber for backup"=38,673,295
INFO [05-15|02:37:58.538] block backup process running successfully "current blockNumber for backup"=38,675,520
INFO [05-15|02:38:03.538] block backup process running successfully "current blockNumber for backup"=38,677,954
INFO [05-15|02:38:08.538] block backup process running successfully "current blockNumber for backup"=38,680,242
INFO [05-15|02:38:13.545] block backup process running successfully "current blockNumber for backup"=38,682,200
INFO [05-15|02:38:18.545] block backup process running successfully "current blockNumber for backup"=38,684,336
INFO [05-15|02:38:23.547] block backup process running successfully "current blockNumber for backup"=38,686,365
INFO [05-15|02:38:28.552] block backup process running successfully "current blockNumber for backup"=38,688,289
INFO [05-15|02:38:33.554] block backup process running successfully "current blockNumber for backup"=38,689,947
INFO [05-15|02:38:38.555] block backup process running successfully "current blockNumber for backup"=38,692,320
INFO [05-15|02:38:43.556] block backup process running successfully "current blockNumber for backup"=38,695,311
INFO [05-15|02:38:48.557] block backup process running successfully "current blockNumber for backup"=38,698,731
INFO [05-15|02:38:53.558] block backup process running successfully "current blockNumber for backup"=38,701,651
INFO [05-15|02:38:58.560] block backup process running successfully "current blockNumber for backup"=38,704,094
INFO [05-15|02:39:03.561] block backup process running successfully "current blockNumber for backup"=38,706,614
INFO [05-15|02:39:08.563] block backup process running successfully "current blockNumber for backup"=38,708,783
INFO [05-15|02:39:13.564] block backup process running successfully "current blockNumber for backup"=38,711,104
INFO [05-15|02:39:18.565] block backup process running successfully "current blockNumber for backup"=38,713,441
INFO [05-15|02:39:23.572] block backup process running successfully "current blockNumber for backup"=38,715,618
INFO [05-15|02:39:28.572] block backup process running successfully "current blockNumber for backup"=38,717,492
INFO [05-15|02:39:33.573] block backup process running successfully "current blockNumber for backup"=38,719,600
INFO [05-15|02:39:38.040] block back up done "current start blockNumber in ancientDB"=38,631,904
INFO [05-15|02:39:38.067] Block pruning BackUp successfully "time duration since start is"=2m59.578s
INFO [05-15|02:39:38.172] Block prune successfully
start geth
➜ geth --config config.toml --history.transactions=0 --syncmode=full --tries-verify-mode=none --db.engine=pebble --datadir geth.fast
t=2024-05-15T03:00:57+0200 lvl=info msg="Use config state scheme" config=hash
t=2024-05-15T03:00:57+0200 lvl=info msg="Automatically disables snap protocol due to verify mode" mode=none
t=2024-05-15T03:00:57+0200 lvl=info msg="Set global gas cap" cap=50000000
t=2024-05-15T03:00:57+0200 lvl=info msg="Initializing the KZG library" backend=gokzg
t=2024-05-15T03:00:57+0200 lvl=info msg="Using pebble as the backing database"
t=2024-05-15T03:00:57+0200 lvl=info msg="Allocated cache and file handles" database=/opt/test/geth.fast/geth/chaindata cache="1.60 GiB" handles=524288 "memory table"="409.50 MiB"
t=2024-05-15T03:00:57+0200 lvl=info msg="Opened ancient database" database=/opt/test/geth.fast/geth/chaindata/ancient/chain readonly=false frozen=38721904
t=2024-05-15T03:00:57+0200 lvl=info msg="State scheme set by user" scheme=hash
t=2024-05-15T03:00:57+0200 lvl=info msg="Allocated trie memory caches" clean="614.00 MiB" dirty="1024.00 MiB"
t=2024-05-15T03:00:57+0200 lvl=info msg=Parlia chainConfig="{ChainID: 56 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: 31302048, YOLO v3: <nil>, CatalystBlock: <nil>, London: 31302048, ArrowGlacier: <nil>, MergeFork:<nil>, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024,Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime: 1705996800, FeynmanTime: 1713419340, FeynmanFixTime: 1713419340, CancunTime: <nil>, Engine: parlia}"
t=2024-05-15T03:00:57+0200 lvl=info msg="Initialising Ethereum protocol" network=56 dbversion=8
t=2024-05-15T03:00:57+0200 lvl=info msg="Initialised chain configuration" config="{ChainID: 56 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: 31302048, YOLO v3: <nil>, CatalystBlock: <nil>, London: 31302048, ArrowGlacier: <nil>, MergeFork:<nil>, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Planck: 27281024,Luban: 29020050, Plato: 30720096, Hertz: 31302048, Hertzfix: 34140700, ShanghaiTime: 1705996800, KeplerTime: 1705996800, FeynmanTime: 1713419340, FeynmanFixTime: 1713419340, CancunTime: <nil>, Engine: parlia}"
t=2024-05-15T03:00:58+0200 lvl=info msg="Loaded most recent local block" number=38722533 hash=0xf9f6ca86c6ee569ff1cb73ff8b7dc8d3725e1d5d55e7a3605e74fb70aac31088 root=0x8f31932a65b59f0ca9cbd8fbc9096140075e048c70e9cd21e771272c5e8eead0 td=76932116 age=6h11m1s
t=2024-05-15T03:00:58+0200 lvl=info msg="Loaded most recent local finalized block" number=38722531 hash=0x2d856153b85bc3b09a8ded03f5ff4059c81583e6db8c81668997a7484c2256c2 root=0xef931cd78d64981d3b9a2f19787ccfec92e7f2efbaa92ebde861ead0097a5753 td=76932113 age=6h11m8s
t=2024-05-15T03:00:58+0200 lvl=info msg="Loaded last snap-sync pivot marker" number=35881922
t=2024-05-15T03:00:58+0200 lvl=warn msg="Head state missing, repairing" number=38722533 hash=0xf9f6ca86c6ee569ff1cb73ff8b7dc8d3725e1d5d55e7a3605e74fb70aac31088 diskRoot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246
t=2024-05-15T03:00:58+0200 lvl=info msg="Snapshot loaded" diskRoot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246 root=0x8f31932a65b59f0ca9cbd8fbc9096140075e048c70e9cd21e771272c5e8eead0
t=2024-05-15T03:00:58+0200 lvl=info msg="Rewound to block with state" number=38722375 hash=0xc8e330e3de53ad2a63985d50da15b196673113b09e66767fdb91801194f677bf
t=2024-05-15T03:00:59+0200 lvl=warn msg="Snapshot is not continuous with chain" snaproot=0x8f31932a65b59f0ca9cbd8fbc9096140075e048c70e9cd21e771272c5e8eead0 chainroot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246
t=2024-05-15T03:00:59+0200 lvl=info msg="Snapshot loaded" diskRoot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246 root=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246
t=2024-05-15T03:00:59+0200 lvl=info msg="Rewound to block with state" number=38722375 hash=0xc8e330e3de53ad2a63985d50da15b196673113b09e66767fdb91801194f677bf
t=2024-05-15T03:00:59+0200 lvl=info msg="Loaded most recent local block" number=38722375 hash=0xc8e330e3de53ad2a63985d50da15b196673113b09e66767fdb91801194f677bf root=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246 td=76931824 age=6h19m8s
t=2024-05-15T03:00:59+0200 lvl=info msg="Loaded most recent local finalized block" number=38722373 hash=0x6a99e96559878c5d77f97cf7e10e4a917105d905ccac1a6335e4bb5d0171c4ed root=0xade8b5fefee38a1b7e7fd33b7a9b41f22ab8382b83199fae187865cbc5cdfeb8 td=76931820 age=6h19m14s
t=2024-05-15T03:00:59+0200 lvl=info msg="Loaded last snap-sync pivot marker" number=35881922
t=2024-05-15T03:00:59+0200 lvl=warn msg="Enabling snapshot recovery" chainhead=38722375 diskbase=38722375
t=2024-05-15T03:01:00+0200 lvl=warn msg="Snapshot is not continuous with chain" snaproot=0x8f31932a65b59f0ca9cbd8fbc9096140075e048c70e9cd21e771272c5e8eead0 chainroot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246
t=2024-05-15T03:01:00+0200 lvl=info msg="Snapshot loaded" diskRoot=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246 root=0xf5ed38b113869315e7b11acea2df2e2c3c726a9e7f2f43d12fe8c43c4b252246
t=2024-05-15T03:01:00+0200 lvl=info msg="Initialized transaction indexer" range="entire chain"
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid txpool lifetime" provided=0s updated=3h0m0s
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid txpool reannounce time" provided=0s updated=1m0s
t=2024-05-15T03:01:00+0200 lvl=info msg="Create votePool successfully"
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid gasprice oracle price cap" provided=<nil> updated=100000000000
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid gasprice oracle ignore price" provided=<nil> updated=4
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid gasprice oracle max header history" provided=0 updated=1
t=2024-05-15T03:01:00+0200 lvl=warn msg="Sanitizing invalid gasprice oracle max block history" provided=0 updated=1
t=2024-05-15T03:01:00+0200 lvl=info msg="Starting peer-to-peer node" instance=Geth/v1.4.6-43b2ffa6-20240429/linux-amd64/go1.22.2
t=2024-05-15T03:01:00+0200 lvl=info msg="New local node record" seq=1715240731947 id=9d1f46dc2faac029cd07165f33e1b806bc8e2d8d3ea07ade5e4e9ac404778ae1 ip=127.0.0.1 udp=30311 tcp=30311
t=2024-05-15T03:01:00+0200 lvl=info msg="Started P2P networking" self=enode://81de309872ef2f11d72f6aeb48744667c37ec6693ca93ba1b638845d4a347f0d8522257b834c90d65724a587e840e03dba26282276606a291b559324582335bd@127.0.0.1:30311
t=2024-05-15T03:01:00+0200 lvl=info msg="IPC endpoint opened" url=/opt/test/geth.fast/geth.ipc
...
t=2024-05-15T03:01:25+0200 lvl=info msg="Imported new chain segment" number=38722382 hash=0xd781576a6f5f8201fc3de05d119700d1e2d0abc7f9aed4d2219311b3c7b62d16 miner=0xfC1004C0f296Ec3Df4F6762E9EabfcF20EB304a2 blocks=7 txs=644 blobs=0 mgas=76.244849 elapsed=1.168s mgasps=65.2735813708198 age=6h19m13s snapdiffs="4.08 MiB" triedirty="0.00 B" trieimutabledirty="0.00 B"
It seems that the problem has been solved, after adding the --block-amount-reserved=90000
parameter
This parameter was not needed before. I decided to open this issue so that people with the same problem can check it.
System information
prunbe command
log
When submitting logs: please submit them as text and not screenshots.