Download snapshots failed

[naviocean]

First of all, Thank you so much for sharing BSC snapshot. Based on your bash script, I've tried to download snapshots but failed.

aws s3 sync --request-payer requester s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/ ./snapshots/

Here is debug information

2022-11-09 06:49:15,078 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.8.10/dist/awscli/botocore/cacert.pem
2022-11-09 06:49:15,079 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): public-blockchain-snapshots.s3.us-west-2.amazonaws.com:443
2022-11-09 06:49:15,768 - MainThread - urllib3.connectionpool - DEBUG - https://public-blockchain-snapshots.s3.us-west-2.amazonaws.com:443 "GET /?list-type=2&prefix=bsc%2Ferigon-snapshots-folder-latest%2F&encoding-type=url HTTP/1.1" 200 None
2022-11-09 06:49:15,768 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'P1FotygX3k8f8GcEoYvW9+opBYqdpqaXY5sZeK1kNM279APSc/gzB9bz5Ah0NXJS1M1eTI8GYFY=', 'x-amz-request-id': 'XMPFC7C0HSNXXACR', 'Date': 'Wed, 09 Nov 2022 06:49:16 GMT', 'x-amz-request-charged': 'requester', 'x-amz-bucket-region': 'us-west-2', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2022-11-09 06:49:15,769 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>public-blockchain-snapshots</Name><Prefix>bsc/erigon-snapshots-folder-latest/</Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>'
2022-11-09 06:49:15,769 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd466675310>>
2022-11-09 06:49:15,769 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-11-09 06:49:15,769 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fd4666753a0>>
2022-11-09 06:49:15,769 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fd46968eca0>
2022-11-09 06:49:15,769 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7fd4677fc700>
2022-11-09 06:49:15,769 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd4666b9dc0>>
2022-11-09 06:49:15,770 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

[allada]

Yes, i am currently in the process of updating all the snapshots in preparation for an Erigon version upgrade. Please try back tomorrow.

[naviocean]

@allada Do I have to download nodes, parlia, and chaindata again?

[allada]

Sadly, probably.

However, because #38 found that the current version that was uploaded is corrupt. There is an interm version that does work that you can use. To download it, you need to run something like:

zfs destroy tank/erigon_data -R -f
s3pcp --requester-pays s3://public-blockchain-snapshots/snap.zfs.zstd | pv | pzstd -p $(nproc) -q -d -f | zfs recv tank/erigon_data

Then mount them with something like:

zfs set mountpoint=/erigon/data tank/erigon_data
zfs set mountpoint=/erigon/data/bsc tank/erigon_data/bsc
zfs set mountpoint=/erigon/data/bsc/chaindata tank/erigon_data/bsc/chaindata
zfs set mountpoint=/erigon/data/bsc/nodes tank/erigon_data/bsc/nodes
zfs set mountpoint=/erigon/data/bsc/parlia tank/erigon_data/bsc/parlia
zfs set mountpoint=/erigon/data/bsc/snapshots tank/erigon_data/bsc/snapshots

Then restart erigon with systemctl start erigon-bsc

This will download and mount all the files in 1 shot.

You'll also need to delete the snapshots it makes or you'll run out of disk space after a while or running it.

See this comment and comments below If you get stuck: https://github.com/allada/bsc-archive-snapshot/issues/38#issuecomment-1306622030

[naviocean]

@allada does the updating snapshot support Erigon v2.29.0?

[allada]

Yes the new version that is uploading now had the database upgraded to v2.29.0. The interm snapshot above was on v2.28.1, which will require an additional few hours of indexing the snapshots folder if you need/want to use v2.29.0.

[naviocean]

@allada thank you so much, I'll try back tomorrow.

[allada]

Closing due to inactivity.