[Project tracking] Archival node database size optimizations

[tayfunelmas]

For details, see this document.

The fine-granular tasks:

• Testing
  • [ ] Implement a neard command to replay the chain from the genesis (related Zulip thread). Note that we do not intend to replay the entire history of mainnet/testnet but smaller networks such as forknet and localnet.
  • [x] Make neard localnet command configure non-validator archival and RPC nodes tracking all shards. [PR]
  • [x] Add a command neard replay-archival to replay the blocks from the genesis block.
  • [ ] Add Nayduck test to test the neard replay-archival command.
  • [ ] Implement an integration test to check JSON-RPC methods (to avoid RPC-breaking changes).
  • [x] Make testloop framework suppport creating ViewClients for issuing JSON-RPC requests.
  • [x] Add one or more testloop tests to check the view client functionality for an archival node.
  • [ ] Support all view-client requests in the testloop test.
• Audit
  • [ ] Instrument code and identify the DB columns needed to replay a localnet from genesis.
• Investigate
  • [ ] How to support providing state view proofs from ReadRPC?

[walnut-the-cat]

Latest update from @tayfunelmas on Zulip:

There are 2 parts:

1. For columns that are not needed in cold storage, I will start marking them as non-Cold and remove from Cold DB. As the first step to this, I am marking PartialChunks as non-Cold in #12029.
2. For the State column, based on the assumption that archival node may not be used for RPC requests, I am thinking about storing only snapshots of State at certain intervals (eg. once per epoch). Then one can replay the chain for any block mid epoch starting from that snapshot. Note that, this will be slow and only for on-demand audit and verification purposes, not for high QPS RPC requests. Regarding how that State snapshot will look like will be depending on some experiments I need to do. For example, it could be some form of Flat state (leaves only but also with some value sharing) or full trie based on the storage each will take.

[walnut-the-cat]

According to SRE team, we have only ~2 months before them having to take an action to deal with ever-increasing size of archival node so this should be considered carefully