S3 permissions request: GetBucketLocation

[mojodna]

Please add s3:GetBucketLocation permission to the bucket policy for the osm-planet-eu-central-1 S3 buckets on AWS. This will allow data consumers to write code that can automatically resolve the right region to use when accessing a bucket w/o parsing the bucket name.

Without this permission, calls will return access denied:

❯ aws s3api get-bucket-location --bucket osm-planet-eu-central-1

An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied

This currently works for the us-west-2-hosted bucket:

❯ aws s3api get-bucket-location --bucket osm-planet-us-west-2
{
    "LocationConstraint": "us-west-2"
}

[tomhughes]

I believe we deliberately direct everybody to the one bucket because we can't guarantee the mirror will work.

[mojodna]

That's a separate issue. I'm asking for the buckets (however they're used) to have matching configurations.

GetBucketLocation is used to determine which region a given bucket is located in, not to do any sort of redirect.

[tomhughes]

Sorry I thought you meant you wanted to get the region in order to decide which bucket to use but you mean you want it in order to decide where to put your consumer?

[mojodna]

Ish. The AWS SDKs require setting a region in order to configure the client that talks to the S3 API. Some S3 APIs (like GetBucketLocation) will work the same regardless of which region is called, but GetObject (which fetches things) requires that the client be configured to talk to the region where the bucket is located, which either needs to be done manually or by using GetBucketLocation. Understandably, many tools use the latter approach.

[Firefishy]

I am not sure why this isn't working, the buckets have identical policies.

osm-planet-eu-central-1 Bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::osm-planet-eu-central-1"
        },
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:GetObjectTagging",
                "s3:GetObjectAttributes",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::osm-planet-eu-central-1/*"
        }
    ]
}

osm-planet-us-west-2 Bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::osm-planet-us-west-2"
        },
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:GetObjectTagging",
                "s3:GetObjectAttributes",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::osm-planet-us-west-2/*"
        }
    ]
}

[Firefishy]

These work: AWS_REGION=eu-central-1 aws s3api --no-sign-request get-bucket-location --bucket osm-planet-eu-central-1 AWS_REGION=us-west-2 aws s3api --no-sign-request get-bucket-location --bucket osm-planet-us-west-2

[mojodna]

I'll do some more digging to figure out how it's supposed to work (to be clear: next action is on me, which may well lead to just closing this issue--thanks!). Something doesn't seem right here (and you shouldn't need to know the bucket's region to, um, look up the bucket's region).

❯ aws --region us-west-2 s3api get-bucket-location --bucket osm-planet-eu-central-1

An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied
❯ aws --region eu-central-1 s3api get-bucket-location --bucket osm-planet-eu-central-1
{
    "LocationConstraint": "eu-central-1"
}

[Firefishy]

I will dig more. Doesn't seem right to me either, but has been reported elsewhere too.

[mojodna]

I'm holding it wrong. I was just looking at https://github.com/aws/aws-sdk-go/issues/720#issuecomment-243891223 too. It sounds like it's related to making calls as someone other than the bucket owner.

This works as expected:

❯ aws --region eu-central-1 --no-sign-request s3api head-bucket --bucket osm-planet-eu-central-1
{
    "BucketRegion": "eu-central-1",
    "AccessPointAlias": false
}
❯ aws --region us-west-2 --no-sign-request s3api head-bucket --bucket osm-planet-eu-central-1
{
    "BucketRegion": "eu-central-1",
    "AccessPointAlias": false
}

Thank you for coming along on my journey.