Is it possible to have a archive node that prunes from time to time?

mikeletux-cube commented 4 months ago

Describe the bug

I'm aware that this is not a bug but I'd like to discuss one scenario that I'm trying to implement in my environment.

Turns out that for our application, we need to select a block hash and from that point onwards our application will be following all blocks from the blockchain.

First of all, we tried with the --pruning=archive flag so that we keep all blockchain history in storage. This is not very convenient for us for two reasons:

We've been having issues syncing an archive node from scratch due to having our node stuck in block 2585476 (as far as I read the problematic block was 2585474). Someone from the Bittensor community team said that a snapshot was gonna to be published to overcome this situation. Do we know when this is going to be published or how to solve the situation? Archive nodes from Bittensor are working but there was no solution for the rest of us.
Also for us is not very convenient to use an archive node because, as I said, past blocks are not important as long as our software follow the chain, so to save storage from time to time we'd like to prune.

We also explored the flags --blocks-pruning 100000 --state-pruning 100000 and so a --sync=warp so that we sync the latest state of the blockchain and we keep the last 100k blocks and states. The thing about this is that, if we stop or restart the service for some maintenance operation, the sync doesn't start where it was left and our software tries to continue also where it left it and complains that block/state that it needs next is not there.

In Ethereum (Geth + lighthouse) it is possible to sync the head of the chain and have an archive node from that point onwards, and if we stop the service for maintenance purposes, when started again, it starts syncing from where it left it, allowing our software to keep up with the chain again. Is there any way of doing this in subtensor/substrate?

Just for your information: we'd like to keep a couple of nodes that are able to sync warp and when synced, keep all future blocks from that point on in local storage, so at some given point we can switch one node off, prune it, and when it's synced up again do the same with the other one, avoiding the need of an full archive node since genesis.

I've read a lot of docs and check also scripts about nodetensor and substrate, but I haven't been able to find a solution for my specific problem.

many thanks in advace!

To Reproduce

Everything explained in Describe the bug

Expected behavior

If possible, do a warp sync of a subtensor node and keep all blocks from the time the node is fully synced.

Screenshots

No response

Environment

Ubuntu 20.04

Additional context

No response

llater commented 4 months ago

This is not a bug

mikeletux-cube commented 4 months ago

That is true, it is not a bug. I think the bug tag is added automatically.

It is a question, but I think it is a genuine one.

Thanks.

opentensor / bittensor