s3_object - ignore_nonexistent_bucket is not used when listing a bucket

Summary

When ansible has the permission to read a subpath of a bucket but not the root path, the list mode fails with a 403 error:

Example:

    local_action:
      module: amazon.aws.aws_s3
      profile: ansible-profile
      bucket: my-bucket
      prefix: allowed/sub/path
      ignore_nonexistent_bucket: yes
      mode: list

Error message: An error occurred (403) when calling the HeadBucket operation: Forbidden",

My suggestion would be to use the validate argument in line 1155 like in line 1070 (got from the ignore_nonexistent_bucket )

By the way, ignore_nonexistent_bucket could be renamed to bypass_permission_control because it is what it really does

Issue Type

Bug Report

Component Name

amazon.aws.aws_s3

Ansible Version

ansible [core 2.13.2]
  config file = /mnt/SystemInfrastructure/ansible/ansible.cfg
  configured module search path = ['/mnt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /mnt/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, May  6 2021, 00:05:59) [GCC 10.2.1 20201203]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

Collection                Version
------------------------- -------
amazon.aws                1.4.0  
ansible.netcommon         1.5.0  
ansible.posix             1.1.1  
ansible.windows           1.4.0  
arista.eos                1.3.0  
awx.awx                   14.1.0 
azure.azcollection        1.4.0  
check_point.mgmt          1.0.6  
chocolatey.chocolatey     1.0.2  
cisco.aci                 1.1.1  
cisco.asa                 1.0.4  
cisco.intersight          1.0.10 
cisco.ios                 1.3.0  
cisco.iosxr               1.2.1  
cisco.meraki              2.2.0  
cisco.mso                 1.1.0  
cisco.nso                 1.0.3  
cisco.nxos                1.4.0  
cisco.ucs                 1.6.0  
cloudscale_ch.cloud       1.3.1  
community.aws             1.3.0  
community.azure           1.0.0  
community.crypto          1.4.0  
community.digitalocean    1.0.0  
community.docker          1.2.2  
community.fortios         1.0.0  
community.general         1.3.6  
community.google          1.0.0  
community.grafana         1.1.0  
community.hashi_vault     1.1.0  
community.hrobot          1.1.0  
community.kubernetes      1.1.1  
community.kubevirt        1.0.0  
community.libvirt         1.0.0  
community.mongodb         1.2.0  
community.mysql           1.2.0  
community.network         1.3.2  
community.okd             1.0.0  
community.postgresql      1.1.1  
community.proxysql        1.0.0  
community.rabbitmq        1.0.1  
community.routeros        1.1.0  
community.skydive         1.0.0  
community.vmware          1.7.0  
community.windows         1.3.0  
community.zabbix          1.2.0  
containers.podman         1.4.1  
cyberark.conjur           1.1.0  
cyberark.pas              1.0.5  
dellemc.os10              1.0.2  
dellemc.os6               1.0.6  
dellemc.os9               1.0.3  
f5networks.f5_modules     1.7.1  
fortinet.fortimanager     1.0.5  
fortinet.fortios          1.1.8  
frr.frr                   1.0.3  
gluster.gluster           1.0.1  
google.cloud              1.0.2  
hetzner.hcloud            1.2.1  
ibm.qradar                1.0.3  
infinidat.infinibox       1.2.4  
junipernetworks.junos     1.3.0  
mellanox.onyx             1.0.0  
netapp.aws                20.9.0 
netapp.elementsw          20.11.0
netapp.ontap              20.12.0
netapp_eseries.santricity 1.1.0  
netbox.netbox             1.2.1  
ngine_io.cloudstack       1.2.0  
ngine_io.exoscale         1.0.0  
ngine_io.vultr            1.1.0  
openstack.cloud           1.2.1  
openvswitch.openvswitch   1.1.0  
ovirt.ovirt               1.3.0  
purestorage.flasharray    1.6.2  
purestorage.flashblade    1.4.0  
servicenow.servicenow     1.0.4  
splunk.es                 1.0.2  
theforeman.foreman        1.5.1  
vyos.vyos                 1.1.1  
wti.remote                1.0.1  

# /mnt/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
amazon.aws      4.1.0  
ansible.utils   2.6.1  
ansible.windows 1.10.0

AWS SDK versions

Name: boto3
Version: 1.24.48
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.48
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: awscli, boto3, s3transfer

Configuration

ANSIBLE_PIPELINING(/mnt/SystemInfrastructure/ansible/ansible.cfg) = True
DEFAULT_HOST_LIST(/mnt/SystemInfrastructure/ansible/ansible.cfg) = ['/mnt/SystemInfrastructure/ansible/inventory']
DEFAULT_PRIVATE_KEY_FILE(env: ANSIBLE_PRIVATE_KEY_FILE) = /mnt/.ssh/id_rsa
DEFAULT_ROLES_PATH(/mnt/SystemInfrastructure/ansible/ansible.cfg) = ['/mnt/SystemInfrastructure/ansible/roles']
DEFAULT_TIMEOUT(/mnt/SystemInfrastructure/ansible/ansible.cfg) = 60

OS / Environment

alpine (docker container)

Steps to Reproduce

name: list s3 bucket local_action: module: amazon.aws.aws_s3 profile: ansible-profile bucket: my-bucket prefix: allowed/sub/path ignore_nonexistent_bucket: yes mode: list register: my_var run_once: true

Expected Results

Be able to list the subpath indicated by prefix.

Actual Results

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.aws_s3_payload_zh3ohb_y/ansible_amazon.aws.aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 481, in bucket_check
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden
fatal: [valid-tst -> localhost]: FAILED! => {
    "boto3_version": "1.24.48",
    "botocore_version": "1.27.48",
    "changed": false,
    "error": {
        "code": "403",
        "message": "Forbidden"
    },
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "bucket": "my-bucket",
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": null,
            "dualstack": false,
            "ec2_url": null,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": true,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "list",
            "object": null,
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "allowed/sub/path",
            "profile": "ansible-profile",
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "rgw": false,
            "s3_url": null,
            "security_token": null,
            "src": null,
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Failed while looking up bucket (during bucket_check) my-bucket.: An error occurred (403) when calling the HeadBucket operation: Forbidden",
    "response_metadata": {
        "host_id": "EhSYJQlLXgTibLCo0ewG0ccAuZBht8uEvG1AisMPo3ar86hHK4YS4fIOqwarhWliFAHQDbeaggA=",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Thu, 11 Aug 2022 09:48:05 GMT",
            "server": "AmazonS3",
            "x-amz-bucket-region": "eu-west-1",
            "x-amz-id-2": "EhSYJQlLXgTibLCo0ewG0ccAuZBht8uEvG1AisMPo3ar86hHK4YS4fIOqwarhWliFAHQDbeaggA=",
            "x-amz-request-id": "018GCMEQPVVANEVP"
        },
        "http_status_code": 403,
        "request_id": "018GCMEQPVVANEVP",
        "retry_attempts": 0
    }
}

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansible-collections / amazon.aws