s3_object: NoSuchTagSet when s3 endpoint does not support tags

Summary

As of Amazon AWS collection v6, the s3_object module fails with NoSuchTagSet when putting an object to an S3 endpoint that does not support object tagging. If I revert to 5.5.0, the same task works without error. Both 6.0.x and 6.1 exhibit the error. The reproduction task below works if I remove the Wasabi endpoint and run against AWS.

Issue Type

Bug Report

Component Name

s3_object

Ansible Version

ansible [core 2.15.0]
  config file = None
  configured module search path = ['/Users/dbarboza/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible
  ansible collection location = /Users/dbarboza/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.3 (main, Apr  7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)] (/usr/local/opt/python@3.11/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

# /Users/dbarboza/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.1.0
ansible.eda                   1.3.8
ansible.netcommon             5.1.1
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
arista.eos                    6.0.1
community.crypto              2.13.1
community.docker              3.4.6
community.general             7.0.1
community.grafana             1.5.4
dellemc.openmanage            7.6.1
kubernetes.core               2.4.0

# /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.0.1
ansible.netcommon             5.1.1
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
arista.eos                    6.0.1
awx.awx                       22.2.0
azure.azcollection            1.15.0
check_point.mgmt              5.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.6.0
cisco.asa                     4.0.0
cisco.dnac                    6.7.2
cisco.intersight              1.0.27
cisco.ios                     4.5.0
cisco.iosxr                   5.0.2
cisco.ise                     2.5.12
cisco.meraki                  2.15.1
cisco.mso                     2.4.0
cisco.nso                     1.0.3
cisco.nxos                    4.3.0
cisco.ucs                     1.8.0
cloud.common                  2.1.3
cloudscale_ch.cloud           2.2.4
community.aws                 6.0.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.13.1
community.digitalocean        1.23.0
community.dns                 2.5.4
community.docker              3.4.6
community.fortios             1.0.0
community.general             7.0.1
community.google              1.0.0
community.grafana             1.5.4
community.hashi_vault         5.0.0
community.hrobot              1.8.0
community.libvirt             1.2.0
community.mongodb             1.5.2
community.mysql               3.7.1
community.network             5.0.0
community.okd                 2.3.0
community.postgresql          2.4.1
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.8.0
community.sap                 1.0.0
community.sap_libs            1.4.1
community.skydive             1.0.0
community.sops                1.6.1
community.vmware              3.6.0
community.windows             1.13.0

AWS SDK versions

Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.26.149
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.149
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/dbarboza/Library/Python/3.11/lib/python/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

CONFIG_FILE() = None
DEPRECATION_WARNINGS(env: ANSIBLE_DEPRECATION_WARNINGS) = True
PAGER(env: PAGER) = less

OS / Environment

MacOS Ventura 13.4

Steps to Reproduce

AWS_PROFILE=wasabi ansible localhost -m amazon.aws.s3_object -a "mode=put endpoint_url=https://s3.wasabisys.com bucket=bucket1 src=/tmp/foo object=foo " -vvv

Note that I changed the profile & bucket names for privacy.

Expected Results

I expected the object to be uploaded to the bucket.

Actual Results

AWS_PROFILE=wasabi ansible localhost -m amazon.aws.s3_object -a "mode=put endpoint_url=https://s3.wasabisys.com bucket=bucket1 src=/tmp/foo object=foo " -vvv
ansible [core 2.15.0]
  config file = None
  configured module search path = ['/Users/dbarboza/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dbarboza/Library/Python/3.11/lib/python/site-packages/ansible
  ansible collection location = /Users/dbarboza/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.3 (main, Apr  7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)] (/usr/local/opt/python@3.11/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
No config file found; using defaults
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: No inventory was parsed, only implicit localhost is available
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: dbarboza
<127.0.0.1> EXEC /bin/sh -c 'echo ~dbarboza && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/dbarboza/.ansible/tmp `"&& mkdir "` echo /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170 `" && echo ansible-tmp-1686243622.235886-25080-67772042048170="` echo /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170 `" ) && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'test -e /tmp/foo && sleep 0'
Using module file /Users/dbarboza/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_object.py
<127.0.0.1> PUT /Users/dbarboza/.ansible/tmp/ansible-local-250430uz258vj/tmpky7sks1f TO /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/ /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@3.11/bin/python3.11 /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/AnsiballZ_s3_object.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/dbarboza/.ansible/tmp/ansible-tmp-1686243622.235886-25080-67772042048170/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1026, in ensure_tags
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 974, in get_current_object_tags_dict
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
           ^^^^^^^^^^^^
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
           ^^^^^^
  File "/Users/dbarboza/Library/Python/3.11/lib/python/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dbarboza/Library/Python/3.11/lib/python/site-packages/botocore/client.py", line 964, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NoSuchTagSet) when calling the GetObjectTagging operation: The TagSet does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1540, in main
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1188, in s3_object_do_put
  File "/var/folders/rl/2llc0bfj09v32zs7dlr9n0hr0000gn/T/ansible_amazon.aws.s3_object_payload_q_z8lkwy/ansible_amazon.aws.s3_object_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1032, in ensure_tags
S3ObjectFailure: Failed to get object tags.
localhost | FAILED! => {
    "boto3_version": "1.26.149",
    "botocore_version": "1.29.149",
    "changed": false,
    "error": {
        "code": "NoSuchTagSet",
        "message": "The TagSet does not exist"
    },
    "invocation": {
        "module_args": {
            "access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "bucket": "bucket1",
            "ceph": false,
            "content": null,
            "content_base64": null,
            "copy_src": null,
            "debug_botocore_endpoint_logs": false,
            "dest": null,
            "dualstack": false,
            "encrypt": true,
            "encryption_kms_key_id": null,
            "encryption_mode": "AES256",
            "endpoint_url": "https://s3.wasabisys.com",
            "expiry": 600,
            "headers": null,
            "ignore_nonexistent_bucket": false,
            "marker": "",
            "max_keys": 1000,
            "metadata": null,
            "mode": "put",
            "object": "foo",
            "overwrite": "different",
            "permission": [
                "private"
            ],
            "prefix": "",
            "profile": "wasabi",
            "purge_tags": true,
            "region": null,
            "retries": 0,
            "secret_key": null,
            "session_token": null,
            "sig_v4": true,
            "src": "/tmp/foo",
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "version": null
        }
    },
    "msg": "Failed to get object tags.: An error occurred (NoSuchTagSet) when calling the GetObjectTagging operation: The TagSet does not exist",
    "response_metadata": {
        "host_id": "ULVVsAhNyIYClSGUxricoaonhDPqHecfHckAwQ2fAWsWHyEqz7css1vdbEPUXIriCSY+O6vsP4av",
        "http_headers": {
            "content-type": "application/xml",
            "date": "Thu, 08 Jun 2023 17:00:23 GMT",
            "server": "WasabiS3/7.14.291-2023-05-15-3fb36ca10e (head12)",
            "transfer-encoding": "chunked",
            "x-amz-id-2": "ULVVsAhNyIYClSGUxricoaonhDPqHecfHckAwQ2fAWsWHyEqz7css1vdbEPUXIriCSY+O6vsP4av",
            "x-amz-request-id": "8E8620150D440987"
        },
        "http_status_code": 404,
        "request_id": "8E8620150D440987",
        "retry_attempts": 0
    }
}

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansible-collections / amazon.aws

s3_object: NoSuchTagSet when s3 endpoint does not support tags #1607