Feature request: ability to prune the old ancient blockchain data

[jsvisa]

Rationale

I'm running a new snap-sync node, downloading a fresh snapshot from https://snapshots.matic.today, after the syncing progress, found the local chaindata consuming 1.1/1.6TB are the ancient data, used toooomuch disk:

$  du --max-depth=1 -h bor/chaindata/
1.1T    bor/chaindata/ancient
1.6T    bor/chaindata/

The old ancient data is useless in most cases, so if we support the ancient data pruning, we can use fewer disks.

Implementation

Seems the binance smartchain has supported this feature(merged in #543) maybe we can backport this feature into go-ethereum.

$ ./bin/bsc snapshot prune-block --help
prune-block [command options]

geth offline prune-block for block data in ancientdb.
The amount of blocks expected for remaining after prune can be specified via block-amount-reserved in this command,
will prune and only remain the specified amount of old block data in ancientdb.
the brief workflow is to backup the the number of this specified amount blocks backward in original ancientdb
into new ancient_backup, then delete the original ancientdb dir and rename the ancient_backup to original one for replacement,
finally assemble the statedb and new ancientDb together.
The purpose of doing it is because the block data will be moved into the ancient store when it
becomes old enough(exceed the Threshold 90000), the disk usage will be very large over time, and is occupied mainly by ancientDb,
so it's very necessary to do block data prune, this feature will handle it.

ETHEREUM OPTIONS:
                                      --datadir value                       Data directory for the databases and keystore (default: "/home/amber/.ethereum")
                                      --datadir.ancient value               Data directory for ancient chain segments (default = inside chaindata, '${datadir}/geth/chaindata/ancient/')
                                      --block-amount-reserved value         Sets the expected remained amount of blocks for offline block prune (default: 0)
                                      --triesInMemory value                 The layer of tries trees that keep in memory (default: 128)
                                      --check-snapshot-with-mpt             Enable checking between snapshot and MPT

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[manav2401]

Hi, pruning is available as a sub command of the new bor cli with bor version v0.3.3. Please refer to the this post for more info (specifically the change log part). Hope this helps.

Also, just fyi, we don't support snap sync yet as it doesn't work out of the box for polygon mainnet. You might want to run it on a full sync node.

[petejkim]

@manav2401 this is different though. geth/bor's pruning is stale state pruning, this is ancient block pruning.

@jsvisa +1, would love to see this (ideally with EIP-4444) implemented in bor

[jsvisa]

@petejkim Sorry, this not fully EIP-4444, it will prune the old historical data only, and not handle the p2p issues.

[petejkim]

Yep, which is why I said "ideally"...because I'd like to see it happen.

[0x090909]

yeah Im also looking for that, lets make it happen

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[jsvisa]

/ping

[temaniarpit27]

Hi @jsvisa The team is currently working on a few high-priority releases, and we couldn't take this up. We will definitely try to look into this soon (I don't have a timeline though, maybe the later half of this month). Thanks for keeping your patience.

[kmalloc]

just have the feeling that this is going to take years to get it working.

[temaniarpit27]

@kmalloc I understand your concerns but I want you to know that we will take this up soon depending on availability. The team is working on some prior commitments.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 28 days with no activity.