s3_bucket fails if bucket does not exist yet

Summary

I am trying to create an S3 bucket on AWS, if it does not exist yet.

This is my task:

- name: Setup S3 bucket
  amazon.aws.s3_bucket:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    region: "{{ aws_region }}"
    name: "{{ s3_bucket_name }}"
    versioning: no
    state: present

It fails if the bucket does not exist yet (according to the docs, a bucket should be created in this case). If the bucket already exists, the task completes successfully.

I suspect that it is due to this change: https://github.com/ansible-collections/amazon.aws/pull/357

For the case of a non-existent bucket, the ListBucket command returns an empty list. The HeadBucket command returns 403 in this case, but the code checks for 404. (Note: I gave my IAM user full admin rights, which include the list-bucket permissions necessary to run HeadBucket).

Would someone be able to confirm that the new code indeed works to create buckets that did not exist before?

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.12.2]
  config file = /Users/thomas/test/playbooks/host1/ansible.cfg
  configured module search path = ['/Users/thomas/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/thomas/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.2 (main, Feb  2 2022, 06:19:27) [Clang 13.0.0 (clang-1300.0.29.3)]
  jinja version = 3.0.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0
ansible.netcommon             2.5.0
ansible.posix                 1.3.0
ansible.utils                 2.4.3
ansible.windows               1.9.0
arista.eos                    3.1.0
awx.awx                       19.4.0
azure.azcollection            1.11.0
check_point.mgmt              2.2.2
chocolatey.chocolatey         1.1.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.6.0
cisco.iosxr                   2.6.0
cisco.ise                     1.2.1
cisco.meraki                  2.6.0
cisco.mso                     1.3.0
cisco.nso                     1.0.3
cisco.nxos                    2.8.2
cisco.ucs                     1.6.0
cloud.common                  2.1.0
cloudscale_ch.cloud           2.2.0
community.aws                 2.2.0
community.azure               1.1.0
community.ciscosmb            1.0.4
community.crypto              2.2.0
community.digitalocean        1.15.0
community.dns                 2.0.6
community.docker              2.1.1
community.fortios             1.0.0
community.general             4.4.0
community.google              1.0.0
community.grafana             1.3.0
community.hashi_vault         2.2.0
community.hrobot              1.2.2
community.kubernetes          2.0.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.3
community.network             3.0.0
community.okd                 2.1.0
community.postgresql          1.6.1
community.proxysql            1.3.1
community.rabbitmq            1.1.0
community.routeros            2.0.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.1
community.windows             1.9.0
community.zabbix              1.5.1
containers.podman             1.9.1
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            4.4.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.14.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.3
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.3
infoblox.nios_modules         1.2.1
inspur.sm                     1.3.0
junipernetworks.junos         2.8.0
kubernetes.core               2.2.3
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.13.0
netapp.elementsw              21.7.0
netapp.ontap                  21.15.1
netapp.storagegrid            21.9.0
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.5.1
ngine_io.cloudstack           2.2.2
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.6.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.12.1
purestorage.flashblade        1.9.0
sensu.sensu_go                1.13.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.27.0
theforeman.foreman            2.2.0
vyos.vyos                     2.6.0
wti.remote                    1.0.3

# /Users/thomas/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
amazon.aws       3.1.1
community.aws    3.1.0
community.docker 2.2.0

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.20.46
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.23.46
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/Users/thomas/test/playbooks/host1/ansible.cfg) = False

OS / Environment

MacOS 12.2.1, ansible installed using brew

Steps to Reproduce

---
# Let's list all buckets with the ListBucket command
- name: List S3 buckets
  community.aws.aws_s3_bucket_info:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    name: "some-bucket-name"
  register: s3_buckets

# Should be an empty list if the bucket does not exist
- name: Print info
  debug:
    msg: "{{ s3_buckets.buckets }}"

# This uses the HeadBucket command and fails
- name: Setup S3 bucket
  amazon.aws.s3_bucket:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECRET_KEY }}"
    region: "{{ aws_region }}"
    name: "some-bucket-name"
    state: present

Expected Results

I expected the task to complete ok.

It works, if the bucket already exists. If fails, if the bucket does not exist yet, although the documentation states that a new bucket should be created.

Actual Results

The full traceback is:
Traceback (most recent call last):
  File "/var/folders/_b/h67l22w15_93p1fj81z8fxfc0000gn/T/ansible_amazon.aws.s3_bucket_payload_mzgkhq4c/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 372, in create_or_update_bucket
  File "/var/folders/_b/h67l22w15_93p1fj81z8fxfc0000gn/T/ansible_amazon.aws.s3_bucket_payload_mzgkhq4c/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 625, in bucket_exists
  File "/usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.20.46",
    "botocore_version": "1.23.46",
    "changed": false,
    "error": {
        "code": "403",
        "message": "Forbidden"
    },
    "invocation": {
        "module_args": {
            "acl": null,
            "aws_access_key": "---CENSORED---",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "ceph": false,
            "debug_botocore_endpoint_logs": true,
            "delete_object_ownership": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "some-bucket-name",
            "object_ownership": null,
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": "eu-west-1",
            "requester_pays": null,
            "s3_url": null,
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "versioning": false
        }
    },
    "msg": "Failed to check bucket presence: An error occurred (403) when calling the HeadBucket operation: Forbidden",
    "resource_actions": [
        "s3:HeadBucket"
    ],
    "response_metadata": {
        "host_id": "cgw0AZfbYormenP0FnQHV947/AnHAvlFsiW8aai4sSCkGjoE66qZVJlc9gg7pfGb3aZwlAL0iKk=",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Fri, 11 Mar 2022 22:10:19 GMT",
            "server": "AmazonS3",
            "x-amz-bucket-region": "eu-west-1",
            "x-amz-id-2": "---CENSORED---",
            "x-amz-request-id": "---CENSORED---"
        },
        "http_status_code": 403,
        "request_id": "ZAWD12VZRHQAGAYH",
        "retry_attempts": 0
    }
}

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansible-collections / amazon.aws