abikouo
commented
4 months ago @mohag it looks like the call to ListBuckets is superfluous when name is provided.
I tried to fix it with #2184, could you please give it a try ?
Open mohag opened 4 months ago
abikouo
commented
4 months ago @mohag it looks like the call to ListBuckets is superfluous when name is provided.
I tried to fix it with #2184, could you please give it a try ?
mohag
commented
4 months ago That does work yes, thanks!
TASK [mongodb : If we have a bucket and no endpoint / region, fetch the endpoint from AWS] ******************************************************************************************************************************************
task path: /home/gertvdb/src/<removed>/ansible/roles/mongodb/tasks/pbm.yml:290
ok: [mongo.qa1.<removed>] => {
"bucket_name": "<removed>-backup-us-east-1-59c5daac",
"buckets": [
{
"bucket_location": {
"LocationConstraint": "us-east-1"
},
"name": "<removed>-backup-us-east-1-59c5daac"
}
],
"changed": false
}
MSG:
Retrieved s3 info.I installed the collection from git+https://github.com/abikouo/amazon.aws.git,s3_buckect_issue_2183
mohag
commented
4 months ago I also have a workaround that requires no bucket permissions / boto3 installation... (S3 redirects to the correct region for a bucket even without permissions, It also returns the correct region in a header, we use that here)
- name: Check the correct region from the "wrong endpoint" error message
ansible.builtin.uri:
url: "https://s3.dualstack.us-east-1.amazonaws.com/{{ backup_s3_bucket }}"
follow_redirects: none
status_code:
- 301 # Wrong region
- 403 # Correct region
register: region_check_request
when:
- backup_s3_endpoint is not defined
- backup_s3_region is not defined
- backup_s3_bucket is defined
- backup_s3_bucket != ''
- name: Get correct region from header on response
ansible.builtin.set_fact:
backup_s3_region: "{{ region_check_request.x_amz_bucket_region }}"
when:
- backup_s3_endpoint is not defined
- backup_s3_region is not defined
- backup_s3_bucket is defined
- backup_s3_bucket != ''
tremble
commented
4 months ago I also have a workaround that requires no bucket permissions / boto3 installation... (S3 redirects to the correct region for a bucket even without permissions, this uses that)
In theory it would be possible to implement all of our modules without technically requiring boto3 (they're just calling an HTTP based API after all...). However, doing so would result in a massive amount of duplicate code. While I can see your use case, and I understand where you're coming from, there comes a point of diminishing returns.
Similarly, when it comes to working around missing permissions on a bucket, I can see the value in making the list_buckets call optional (technically it's not superfluous, it's returning creation_date too), but hacking around missing GetBucketLocation is something to poke Amazon about, not something we should be adding here. There's also no guarantee that at some point Amazon won't close that loophole.
mohag
commented
4 months ago I also have a workaround that requires no bucket permissions / boto3 installation... (S3 redirects to the correct region for a bucket even without permissions, this uses that)
In theory it would be possible to implement all of our modules without technically requiring boto3 (they're just calling an HTTP based API after all...). However, doing so would result in a massive amount of duplicate code. While I can see your use case, and I understand where you're coming from, there comes a point of diminishing returns.
Similarly, when it comes to working around missing permissions on a bucket, I can see the value in making the list_buckets call optional (technically it's not superfluous, it's returning creation_date too), but hacking around missing
GetBucketLocationis something to poke Amazon about, not something we should be adding here. There's also no guarantee that at some point Amazon won't close that loophole.
I agree that that workaround does not make sense in the module, but it might be useful for someone else (and I have somewhere that I can find it was well in the future) (It was mainly my solution before the change to get rid of ListBuckets) (I currently install boto3 / botocore on the managed nodes just for this one step, so I'm thinking of keeping it)
Edit: The updated version no longer need xmltodict (It uses a header instead of parsing the response)
Summary
When using aws_s3_bucket_info with a supplied bucket name to determine the bucket location, an unnecessary ListBuckets call means that more permissions are needed than what is needed to determine a bucket's location.
get_bucket_listalways call list_buckets, even if a bucket name is given, which fails if the IAM role / user does not have ListBuckets permissions (but has sufficient permissions to get the requested info)(Yes, I'm likely on some unsupported versions, but the logic in the code will still have the same results on up to date versions)
Issue Type
Bug Report
Component Name
s3_bucket_info
Ansible Version
Collection Versions
AWS SDK versions
Configuration
(Versions retrieved from inventory host, since boto is needed there. AWS CLI was installed after the run, before the version check (to get the expected results)
OS / Environment
Ubuntu 20.04 on WSL1
Steps to Reproduce
Bucket policy:
Expected Results
AWS CLI running with the same credentails (the instance's role) (the region in the name is not always accurate)
The role does not have permission to use ListBuckets:
Actual Results
Code of Conduct