search

ansible-collections / amazon.aws

Ansible Collection for Amazon AWS
GNU General Public License v3.0
304 stars 332 forks source link

ResourceNotFoundException is raised when querying ACM certificates #622

Closed sebastien-rosset closed 2 years ago

sebastien-rosset commented 2 years ago

Summary

A ResourceNotFoundException may be raised when a ansible module invokes the get_certificates function defined in module_utils/acm.py. It looks like there is an intermittent timing issue:

  1. The get_certificates function gets all certificates defined in ACM using paginated queries.
  2. Iterate over every certificate ARN obtained from step 1, and get additional data about the certificate. However some certs may have been deleted between steps 1 and 2, causing a ResourceNotFoundException.

Issue Type

Bug Report

Component Name

acm in module_utils

Ansible Version

$ ansible --version
nsible [core 2.12.1]
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/.local/lib/python3.8/site-packages/ansible
  ansible collection location =~/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible
  python version = 3.8.11 (default, Sep  1 2021, 12:33:46) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
  jinja version = 3.0.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
# ~/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
amazon.aws       3.0.0  
community.aws    2.1.0  
community.crypto 2.1.0  

# ~/.local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

# /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: ~/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.20.21
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, botocore, s3transfer
Required-by: 
---
Name: botocore
Version: 1.23.21
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, urllib3, python-dateutil
Required-by: s3transfer, boto3

Configuration

$ ansible-config dump --only-changed

OS / Environment

CentOS Linux release 7.9.2009 (Core) Linux 5ada6014185e 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Steps to Reproduce

The issue occurs intermittently when running the integration tests for the acm module in community.aws. Specifically when a community.aws.aws_acm task deletes a cert, followed by the community.aws.aws_acm_info. The ResourceNotFoundException occurs in the aws_acm_info task.

  - name: delete cert
    community.aws.aws_acm:
      state: absent
      domain_name: 'mydoamain.com'
  - name: check cert was deleted
    community.aws.aws_acm_info:
      tags:
        Name: 'my_certificate_name'

The problem occurs when a cert is removed from ACM while acm.py:get_certificates iterates a list of cert ARNs. The problem is more likely to occur if the acm.py:get_certificates takes a long time to iterate over the certificate ARNs. For example this could happen if there are lots of certificates in ACM.

If there are N certificates in ACM, get_certificates could send up to 1 + 3 * N requests to the AWS API, not counting the retries (for each cert, get_certificates makes up to 2 or 3 AWS API calls depending on the cert status.)

Expected Results

I was expecting community.aws.aws_acm_info to return a list of certificates successfully, even if certificates are being deleted while querying the certs. Internally the aws_acm_info module invokes the amazon.aws:get_certificates function which is defined in module_utils/acm.py. That function should skip certificates when a ResourceNotFound exception is raised. I.e. don't return certs that are being deleted. The problem exists in the main branch.

Actual Results

An exception is raised while executing the aws_acm_info task.

TASK [aws_acm : check cert 3 was deleted] ********************************************************************************************************************
task path: /root/ansible_collections/community/aws/tests/output/.tmp/integration/aws_acm-uxj6vjlf-ÅÑŚÌβŁÈ/tests/integration/targets/aws_acm/tasks/main.yml:435
Using module file /root/ansible_collections/community/aws/plugins/modules/aws_acm_info.py
Pipelining is enabled.
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /usr/bin/python3.8 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 111, in get_certificates
    cert_data = self.describe_certificate_with_backoff(client, certificate['CertificateArn'])
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 118, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 82, in describe_certificate_with_backoff
    return client.describe_certificate(CertificateArn=certificate_arn)['Certificate']
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the DescribeCertificate operation: Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXXX.
fatal: [testhost]: FAILED! => {
    "boto3_version": "1.16.0",
    "botocore_version": "1.19.0",
    "changed": false,
    "error": {
        "code": "ResourceNotFoundException",
        "message": "Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXX.”
    },

Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help