erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.09k stars 1.08k forks source link

add all snapshots to manifest.txt #11321

Closed Giulio2002 closed 1 month ago

Giulio2002 commented 1 month ago

Rationale

This will allow the implementation of an extremely fast sync up routine for Erigon 3.0.

What is this.

If we look at an Erigon snapshot dir we can see the following content (i just used the content of the folder domain for simplicity:

v1-accounts.0-128.bt          v1-accounts.128-136.bt    v1-code.0-128.kv          v1-code.128-136.kv      v1-commitment.0-128.kv.torrent  v1-commitment.128-136.kvei   v1-storage.0-128.kvei
v1-accounts.0-128.kv          v1-accounts.128-136.kv    v1-code.0-128.kv.torrent  v1-code.128-136.kvei    v1-commitment.0-128.kvei        v1-storage.0-128.bt          v1-storage.128-136.bt
v1-accounts.0-128.kv.torrent  v1-accounts.128-136.kvei  v1-code.0-128.kvei        v1-commitment.0-128.bt  v1-commitment.128-136.bt        v1-storage.0-128.kv          v1-storage.128-136.kv
v1-accounts.0-128.kvei        v1-code.0-128.bt          v1-code.128-136.bt        v1-commitment.0-128.kv  v1-commitment.128-136.kv        v1-storage.0-128.kv.torrent  v1-storage.128-136.kvei

As you can see there are some files which have a .torrent and some others that do not, the files who have a .torrent are displayed in the manifest.txt of the snapshots code and are webseeded to the torrent, the others are not.

The feauture would be to include in the manifest.txt the files who are also NOT in possess of a .torrent, and also serve them. and then update the bucket once every 7 days circa. (Naturally we will end up with some files that do not a .torrent and others who d0).

This should be done for ALL snapshots transactions, bodies, etc...

Giulio2002 commented 1 month ago

for example - v1-storage.128-136.kv does not have a .torrent...

michelemodolo commented 1 month ago

Done.