(HTTP 403) failed to download the s3 files

[jsvisa]

First of all, thank you very much for providing this project.

I'm running this script in region ap-southeast-1's EC2, and with AWS S3 configured. But it failed to download the files, the error code as below:

$ aws s3 ls --request-payer=requester s3://public-blockchain-snapshots/bsc/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
$ export S3_BUCKET_PATH="s3://public-blockchain-snapshots"
$ aws s3 cp --request-payer=requester "$S3_BUCKET_PATH/support/zstd-v1.5.0-linux-x86.tar.gz" .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Could you please help me to find out what's the problem?

Thanks a lot.

[allada]

I suggest trying in us-west-2 (as it's pretty much free to download) and if it works there then do your testing in your preferred data center.

Can you run:

aws sts get-caller-identity

And make sure it succeeds (you don't need to post results here). If it does not succeed, it means that you don't have an AWS role setup on that instance.

I suspect the issue is that the instance is either missing a role or the role it has doesn't have permissions to be changed for requester pays configuration. The easiest way to test this is to put in your personal AWS credentials on the instance using aws configure then try, if it works, it's a role issue.

I have was helping out another company at one point and had ssh access into the instance and was able to download it a few months ago and have changed no permission stuff, so I am pretty sure it should be fine on my end.

Let me know, -Allada

[jsvisa]

Thanks for your reply, I’ll try your instructions later today.

BTW, what’s the type of ec2 instance are you running the archive node? I’m running in c5.x4large with gp3 disk to running the bsc archive node, but the sync status showed the bsc node always lags about 3weaks On Sep 5, 2021, 06:06 +0800, Nathan (Blaise) Bruer @.***>, wrote:

I suggest trying in us-west-2 (as it's pretty much free to download) and if it works there then do your testing in your preferred data center. Can you run: aws sts get-caller-identity And make sure it succeeds (you don't need to post results here). If it does not succeed, it means that you don't have an AWS role setup on that instance. I suspect the issue is that the instance is either missing a role or the role it has doesn't have permissions to be changed for requester pays configuration. The easiest way to test this is to put in your personal AWS credentials on the instance using aws configure then try, if it works, it's a role issue. I have was helping out another company at one point and had ssh access into the instance and was able to download it a few months ago and have changed no permission stuff, so I am pretty sure it should be fine on my end. Let me know, -Allada — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[allada]

My use case is a bit different, I rarely need to run the entire archive at a time. Instead I usually only run the last few months worth, so I use smaller instances and only run the service on a subset of the snapshots.

When I need the entire chain I usually use i3en.12xlarge or i3en.24xlarge as spot instances, since I am not worried if they go down unexpectedly (due to my use case).

I would not suggest using EBS volumes, they are extremely slow and it is often cheaper to use instances with ephemeral storage than smaller instances with EBS.

If cost is really a concern and you don't need it to be super fast, the version of geth I provided in the install script has modified the timeout of calls. You can just use git diff to find out what the changes are. If you increase it even more, I have had success using the d3en class of instances with much smaller EBS (configured as io2) and then use ZFS and configure it to be an L2ARC. You can configure those HDD's to be RIAD0'ed together. I haven't tested this in a long time, but I was able to keep up with the chain when I tested back in May.

[mnguyen-io]

Hi, not sure if you got this working @jsvisa - but I also get the 403 error. Even from us-west-2

# aws configure
AWS Access Key ID [None]:ACCESSKEYID
AWS Secret Access Key [None]: SECRETKEY
Default region name [None]: us-west-2
Default output format [None]: 

# aws sts get-caller-identity
{
    "UserId": "XXXXX",
    "Account": "7011111111111",
    "Arn": "arn:aws:iam::7011111111111:user/user"
}

# aws s3 ls --request-payer=requester s3://public-blockchain-snapshots/bsc/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

# aws s3 cp --request-payer=requester  s3://public-blockchain-snapshots/bsc/erigon-latest.tar.zstd .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Any help would be appreciated @allada

[allada]

Is it possible that the AWS user you are using is not allowed to be charged for S3 actions from the account owner it's under?

I have reports of other users being able to use it without issue just yesterday.

[mnguyen-io]

Owner

allada commented 1 hour ago •

Ah you're absolutely correct - thank you very much