Configure S3 Cross-Region Replication

[DefiDebauchery]

As we're all painfully aware, BSC snapshots are currently only hosted in AWS' Asia-Pacific region.

AWS has a Cross-Region Replication (CRR) feature that would make this data available in more local zones.

This should be done in addition to distribution via torrent, as requested in #26, and would likely make torrent availability more feasible, as volunteers would have much easier access to the source data.

[iakisme]

@DefiDebauchery Cool, let me try, make torrent is not a good idea.

[DefiDebauchery]

I look forward to the result of your tests!

Can you elaborate on why you feel torrents would not be a good idea? It would reduce the transfer costs for Binance's AWS account and increase availability. If you are the one creating the torrent, data integrity would be preserved and enforced; nobody would be able to forge the chain state, if that is your worry.

[DefiDebauchery]

Hi @iakisme - Have you been able to look into this further? I see you've made a couple of new snapshots since your reply.

With the public, consolidated RPC endpoints being so overloaded, allowing people to run their own nodes is more important than ever before; giving access to this data quickly is imperative for that goal.

[bitkin]

Hi @iakisme I'm trying to download the snapshot in the Ireland region of the AWS instance but it is taking too much time. Can you enable the Transfer Acceleration or Cross-Region Replication or any CDN so that the file can be downloaded in feasible time. I tried the larger instance size but it is taking more than 20 hours to download the file. Please help!

[iakisme]

can you try this download link? @DefiDebauchery @bitkin geth-20211101.tar.gz

[bitkin]

@iakisme Thank you so much. It's a great help! :)

[bitkin]

@iakisme The file is downloaded but I think it's not complete. The size was small and when I extracted, it said unexpected end of file. The download file is about 372 GB in size. Can you please check

[mroccon]

@iakisme Looks like the token for the EU endpoint is expired, can you please fix it? Thanks

[iakisme]

done

[iakisme]

@bitkin We pruned the data , and the size is current.

[vladislavabramov]

Hey @iakisme Getting same issue as @bitkin, has it been fixed along with the endpoint fix?

... server/data-seed/geth/chaindata/ancient/bodies.0104.cdat server/data-seed/geth/chaindata/ancient/receipts.0085.cdat server/data-seed/geth/chaindata/ancient/bodies.0105.cdat

gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now

[mjay0]

can also confirm that the EU snapshot is corrupt.

[Lajoix]

can also confirm that the EU snapshot is corrupt.

It is for me also

[nftgeek]

@iakisme Data corrupted!

[iakisme]

@nftgeek @Lajoix @vladislavabramov sorry for that, please try the lastest snapshot