search

ansible-collections / community.aws

Ansible Collection for Community AWS
GNU General Public License v3.0
187 stars 395 forks source link

s3_logging throws XML error in 3.0.1 #912

Closed simon3270 closed 2 years ago

simon3270 commented 2 years ago

Summary

When I try to set up server logging from a new S3 bucket, an XML error is thrown in v3.0.1. Exactly the same code works when I ansible-galaxy install v2.2.0.

The call below follows a successful call to aws_s3_bucket_info.

Issue Type

Bug Report

Component Name

s3_logging

Ansible Version

$ ansible --version
ansible 2.10.10
  config file = /var/lib/jenkins/jobs/cluster-destroy-cmp-branch/workspace/ansible/ansible.cfg
  configured module search path = ['/home/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible
  python version = 3.8.6 (default, Oct 27 2020, 09:13:12) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]

Collection Versions

$ ansible-galaxy collection list
Collection                Version
------------------------- -------
amazon.aws                1.4.0  
ansible.netcommon         1.5.0  
ansible.posix             1.1.1  
ansible.windows           1.4.0  
arista.eos                1.3.0  
awx.awx                   14.1.0 
azure.azcollection        1.4.0  
check_point.mgmt          1.0.6  
chocolatey.chocolatey     1.0.2  
cisco.aci                 1.1.1  
cisco.asa                 1.0.4  
cisco.intersight          1.0.10 
cisco.ios                 1.3.0  
cisco.iosxr               1.2.1  
cisco.meraki              2.2.0  
cisco.mso                 1.1.0  
cisco.nso                 1.0.3  
cisco.nxos                1.4.0  
cisco.ucs                 1.6.0  
cloudscale_ch.cloud       1.3.1  
community.aws             1.3.0  
community.azure           1.0.0  
community.crypto          1.4.0  
community.digitalocean    1.0.0  
community.docker          1.2.2  
community.fortios         1.0.0  
community.general         1.3.6  
community.google          1.0.0  
community.grafana         1.1.0  
community.hashi_vault     1.1.0  
community.hrobot          1.1.0  
community.kubernetes      1.1.1  
community.kubevirt        1.0.0  
community.libvirt         1.0.0  
community.mongodb         1.2.0  
community.mysql           1.2.0  
community.network         1.3.2  
community.okd             1.0.0  
community.postgresql      1.1.1  
community.proxysql        1.0.0  
community.rabbitmq        1.0.1  
community.routeros        1.1.0  
community.skydive         1.0.0  
community.vmware          1.7.0  
community.windows         1.3.0  
community.zabbix          1.2.0  
containers.podman         1.4.1  
cyberark.conjur           1.1.0  
cyberark.pas              1.0.5  
dellemc.os10              1.0.2  
dellemc.os6               1.0.6  
dellemc.os9               1.0.3  
f5networks.f5_modules     1.7.1  
fortinet.fortimanager     1.0.5  
fortinet.fortios          1.1.8  
frr.frr                   1.0.3  
gluster.gluster           1.0.1  
google.cloud              1.0.2  
hetzner.hcloud            1.2.1  
ibm.qradar                1.0.3  
infinidat.infinibox       1.2.4  
junipernetworks.junos     1.3.0  
mellanox.onyx             1.0.0  
netapp.aws                20.9.0 
netapp.elementsw          20.11.0
netapp.ontap              20.12.0
netapp_eseries.santricity 1.1.0  
netbox.netbox             1.2.1  
ngine_io.cloudstack       1.2.0  
ngine_io.exoscale         1.0.0  
ngine_io.vultr            1.1.0  
openstack.cloud           1.2.1  
openvswitch.openvswitch   1.1.0  
ovirt.ovirt               1.3.0  
purestorage.flasharray    1.6.2  
purestorage.flashblade    1.4.0  
servicenow.servicenow     1.0.4  
splunk.es                 1.0.2  
theforeman.foreman        1.5.1  
vyos.vyos                 1.1.1  
wti.remote                1.0.1

AWS SDK versions

$ pip show boto boto3 botocore
       "Name: boto",
       "Version: 2.49.0",
       "Summary: Amazon Web Services Library",
       "Home-page: https://github.com/boto/boto/",
       "Author: Mitch Garnaat",
       "Author-email: mitch@garnaat.com",
       "License: MIT",
       "Location: /usr/local/lib/python3.6/site-packages",
       "Requires: ",
       "---",
       "Name: boto3",
       "Version: 1.17.69",
       "Summary: The AWS SDK for Python",
       "Home-page: https://github.com/boto/boto3",
       "Author: Amazon Web Services",
       "Author-email: None",
       "License: Apache License 2.0",
       "Location: /usr/local/lib/python3.6/site-packages",
       "Requires: jmespath, botocore, s3transfer",
       "---",
       "Name: botocore",
       "Version: 1.20.112",
       "Summary: Low-level, data-driven core of boto 3.",
       "Home-page: https://github.com/boto/botocore",
       "Author: Amazon Web Services",
       "Author-email: None",
       "License: Apache License 2.0",
       "Location: /usr/local/lib/python3.6/site-packages",
       "Requires: jmespath, urllib3, python-dateutil"

Configuration

$ ansible-config dump --only-changed
ansible-config 2.10.10
  config file = /var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg
  configured module search path = ['/home/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible-config
  python version = 3.8.6 (default, Oct 27 2020, 09:13:12) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
Using /var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg as config file
ANSIBLE_FORCE_COLOR(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = True
ANSIBLE_PIPELINING(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=600s
ANSIBLE_SSH_CONTROL_PATH(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = %(directory)s/%%h-%%r
CACHE_PLUGIN(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = .ansible/facts
CACHE_PLUGIN_TIMEOUT(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = 600
COLOR_CHANGED(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = yellow
COLOR_DEBUG(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = dark gray
COLOR_DEPRECATE(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = purple
COLOR_DIFF_ADD(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = green
COLOR_DIFF_LINES(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = cyan
COLOR_DIFF_REMOVE(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = red
COLOR_ERROR(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = red
COLOR_OK(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = green
COLOR_SKIP(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = cyan
COLOR_UNREACHABLE(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = red
COLOR_VERBOSE(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = blue
COLOR_WARN(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = bright purple
DEFAULT_FORKS(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = 20
DEFAULT_GATHERING(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = smart
DEFAULT_REMOTE_USER(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = ****
DEFAULT_ROLES_PATH(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = ['/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/roles']
DEFAULT_VERBOSITY(env: ANSIBLE_VERBOSITY) = 3
HOST_KEY_CHECKING(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = False
RETRY_FILES_ENABLED(/var/lib/jenkins/jobs/build-cmp-branch/workspace/ansible/ansible.cfg) = False

OS / Environment

RHEL 8.4

Steps to Reproduce

    - name: Add Server Access Logging for bucket, branch builds
      s3_logging:
        name: "{{ image_bucket.name }}"
        target_bucket: "my-bucket-s3-server-access-logging"
        target_prefix: "{{ s3_object_prefix }}-{{ cluster_name }}/"
        state: present

Expected Results

I expected sever logging to be enabled to the specified target bucket and logging prefix.

Actual Results

redirecting (type: modules) ansible.builtin.s3_logging to community.aws.s3_logging
Using module file /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible_collections/community/aws/plugins/modules/s3_logging.py
Pipelining is enabled.
<10.64.168.179> ESTABLISH SSH CONNECTION FOR USER: ****
<10.64.168.179> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="****"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="****"' -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/%h-%r 10.64.168.179 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<10.64.168.179> (1, b'\n{"failed": true, "msg": "S3ResponseError: 400 Bad Request\\n<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<Error><Code>MalformedACLError</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>3X1XBPVX559JD0VR</RequestId><HostId>2xgdnO0IVc+kSsgmFg8ZXrHi59ZkeStbX2sd6sKOT0PE2+BcQjFpIzeTzb/JvkalN+IP7rVV+cQ=</HostId></Error>", "exception": "  File \\"/tmp/ansible_s3_logging_payload_vw52n831/ansible_s3_logging_payload.zip/ansible_collections/community/aws/plugins/modules/s3_logging.py\\", line 106, in enable_bucket_logging\\n  File \\"/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py\\", line 1240, in set_as_logging_target\\n    self.set_acl(policy, headers=headers)\\n  File \\"/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py\\", line 942, in set_acl\\n    headers, version_id)\\n  File \\"/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py\\", line 937, in set_xml_acl\\n    response.status, response.reason, body)\\n", "invocation": {"module_args": {"name": "bld-80-fjzct-image-registry-eu-west-1", "target_bucket": "my-bld-s3-server-access-logging", "target_prefix": "image-registry-bld-80a/", "state": "present", "debug_botocore_endpoint_logs": false, "validate_certs": true, "ec2_url": null, "aws_access_key": null, "aws_secret_key": null, "security_token": null, "aws_ca_bundle": null, "profile": null, "aws_config": null, "region": null}}}\n', b'')
<10.64.168.179> Failed to connect to the host via ssh: 
The full traceback is:
  File "/tmp/ansible_s3_logging_payload_vw52n831/ansible_s3_logging_payload.zip/ansible_collections/community/aws/plugins/modules/s3_logging.py", line 106, in enable_bucket_logging
  File "/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py", line 1240, in set_as_logging_target
    self.set_acl(policy, headers=headers)
  File "/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py", line 942, in set_acl
    headers, version_id)
  File "/usr/local/lib/python3.6/site-packages/boto/s3/bucket.py", line 937, in set_xml_acl
    response.status, response.reason, body)
fatal: [10.64.168.179]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "debug_botocore_endpoint_logs": false,
            "ec2_url": null,
            "name": "bld-80-fjzct-image-registry-eu-west-1",
            "profile": null,
            "region": null,
            "security_token": null,
            "state": "present",
            "target_bucket": "my-bld-s3-server-access-logging",
            "target_prefix": "image-registry-bld-80a/",
            "validate_certs": true
        }
    },
    "msg": "S3ResponseError: 400 Bad Request\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MalformedACLError</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>3X1XBPVX559JD0VR</RequestId><HostId>2xgdnO0IVc+kSsgmFg8ZXrHi59ZkeStbX2sd6sKOT0PE2+BcQjFpIzeTzb/JvkalN+IP7rVV+cQ=</HostId></Error>"

Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 2 years ago

cc @jillr @markuman @s-hertel @tremble click here for bot help

tremble commented 2 years ago

@simon3270,

Thanks for taking the time to open this issue.

Looking at the output of ansible-galaxy collection list that you provided it looks like you're using a much older version of the collections than you think you are. Specifically:

amazon.aws                1.4.0  
community.aws             1.3.0

The stack trace you provided also mentions site-packages/boto/s3/bucket.py this is the original AWS Python SDK (boto). With community.aws 1.5.0 we removed the last of the code from s3_logging which used the boto SDK, migrating over to boto3/botocore. This would reinforce the implication that you're not using 3.0.1.

Please could you try re-installing the collection to update the version of the code you're using.

simon3270 commented 2 years ago

Thanks for the quick reply, @tremble :-)

My original code worked fine up until 24th January but then started throwing the XML error. This was about when 3.0.0 dropped, so my initial thought was that the 3.x.x version was a problem, so try the latest 2.x.x one (2.2.0). When I had added a call to ansible-galaxy collection install -r requirements.yml, where requirements.yml had community.aws in it at version 2.2.0, the code worked. I now think that it worked not because I was avoiding 3.x.x, but because I was no longer using the old 1.3.0! To test this, I tried again with 3.0.1 and all was OK.

With 3.0.1 being used, the collection list looks like this, with the extra local section at the top:

+ ansible-galaxy collection list

# /home/jenkins/.ansible/collections/ansible_collections
Collection    Version
------------- -------
amazon.aws    3.0.0  
community.aws 3.0.1  

# /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible_collections
Collection                Version
------------------------- -------
amazon.aws                1.4.0  
...
community.aws             1.3.0

I think we can now close this issue, and I'll get back to my Jenkins administrators and try to get a more up-to-date base image to get more recent versions of some of the other collections.

Thanks again, Simon