Evergreen files

[AskAlexSharov]

Checklist:

• [x] to store mainnet.toml on R2 (done: see .chain.toml)
• [x] decide whether we store .chain.toml on R2 or on raw.github (attempt to use raw.github was here: https://github.com/ledgerwatch/erigon-snapshot/pull/155 ). pros of raw.github - support of git branches, easier release/rollback
• [x] decide - whether .chain.toml must be updated before or after merge PR to https://github.com/ledgerwatch/erigon-snapshot
• [x] release/rollback steps of .chain.toml in case of problem
• [x] what if node started, read .chain.toml and then .chain.toml updated (before downloading finish). will node stuck?
• [x] support .chain.toml from erigon side
• [x] what if r2 is not available. what are fallback steps of erigon node (to compiled-in version of .chain.toml? wait? something else?)
• [x] how to make network more self-sustaining: enough seeds for new .kv files, enough seeds for old .kv files
• [x] decide how long will .kv files live in R2 bucket. Current TTL is 8weeks.
• [x] test upgrade/downgrade of erigon version

Sub-tasks:

• [x] https://github.com/erigontech/erigon/issues/11409
• [x] https://github.com/erigontech/erigon/issues/11410

[mh0lt]

I'd like to suggest an alternative to using r2 to distributing chain contents.

I think we should do the following:

1. Publish the chain.toml files as first class torrents which can retried using a magnet-link, or just the info hash- which is unique.
2. Locate the appropriate link using DNS

the rational for this is that it removes the R2 dependency and has a torrent based fail-over as other peers with the chain file can server it.

I think we can use our snapshotters as the primary provider for the torrents, with a back-up on R2. This makes the model consistent with all other files.

An example of how this may look using info hash ant txt records, (we may also be able to use URL records - but this would need more investigation)

latest._chain.mainnet.erigon.network. CNAME 1.20240128._chain.mainnet.erigon.network.
0.20240127._chain.mainnet.erigon.network. IN TXT "5623d8d6aa7f3aca0d034fd2345ba1d64202145e" 
0.20240128._chain.mainnet.erigon.network. IN TXT "8972b8d6aa7f3aca0d034fd2345ba1d6445da4a4" 
1.20240128._chain.mainnet.erigon.network. IN TXT "7319f8d6aa7f3aca0d034fd2345ba1d6429dc37f" 

or

latest._chain.mainnet.erigon.network. IN TXT "v=202401281"
_chain.mainnet.erigon.network. IN TXT "v=202401270;l=5623d8d6aa7f3aca0d034fd2345ba1d64202145e" 
_chain.mainnet.erigon.network. IN TXT "v=202401280;l=8972b8d6aa7f3aca0d034fd2345ba1d6445da4a4" 
_chain.mainnet.erigon.network. IN TXT "v=202401281;l=7319f8d6aa7f3aca0d034fd2345ba1d6429dc37f" 

For security and integrity we would likely need to enable DNSSEC on our domains - then the client can check that the record is valid as it is signed by us.

If you assume the above the downloader start up would be as follows:

1. Using the chain name do a DNS record look up for the chain subdomain records. Pick the record you want, mainly latest, pr a version you previously downloaded.
2. Download the chain.toml from the torrent. (Note this includes webseed location)
3. Download as before.