search

ansible-collections / community.aws

Ansible Collection for Community AWS
GNU General Public License v3.0
189 stars 398 forks source link

community.aws.rds_cluster module fails to manage cluster after creating cluster #1409

Closed imesias closed 2 years ago

imesias commented 2 years ago

Summary

When I try and create a cluster using community.aws.rds_cluster to create an rds cluster. Upon first execution of the playbook the cluster is created. When run again expecting to see 'ok' instead of changed the module fails with an error:

Traceback (most recent call last):
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.aws.plugins.modules.rds_cluster', init_globals=dict(_module_fqn='ansible_collections.community.aws.plugins.modules.rds_cluster', _modlib_path=modlib_path),
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1026, in <module>
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1010, in main
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 872, in ensure_present
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 846, in changing_cluster_options
KeyError: 'DBClusterParameterGroupName'

Issue Type

Bug Report

Component Name

rds_cluster

Ansible Version

$ ansible --version
ansible [core 2.13.2]
  config file = /Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg
  configured module search path = ['/Users/USERNAME/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/USERNAME/.local/share/virtualenvs/proj-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/USERNAME/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/USERNAME/.local/share/virtualenvs/proj-iac-ansible-aws-m1mSr6qK/bin/ansible
  python version = 3.9.13 (main, May 24 2022, 21:28:44) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    3.3.1  
ansible.netcommon             3.0.1  
ansible.posix                 1.4.0  
ansible.utils                 2.6.1  
ansible.windows               1.10.0 
arista.eos                    5.0.1  
awx.awx                       21.4.0 
azure.azcollection            1.13.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.3.0  
cisco.aci                     2.2.0  
cisco.asa                     3.1.0  
cisco.dnac                    6.5.2  
cisco.intersight              1.0.19 
cisco.ios                     3.2.0  
cisco.iosxr                   3.2.0  
cisco.ise                     2.5.0  
cisco.meraki                  2.10.1 
cisco.mso                     2.0.0  
cisco.nso                     1.0.3  
cisco.nxos                    3.1.0  
cisco.ucs                     1.8.0  
cloud.common                  2.1.2  
cloudscale_ch.cloud           2.2.2  
community.aws                 3.4.0  
community.azure               1.1.0  
community.ciscosmb            1.0.5  
community.crypto              2.4.0  
community.digitalocean        1.21.0 
community.dns                 2.3.0  
community.docker              2.7.0  
community.fortios             1.0.0  
community.general             5.4.0  
community.google              1.0.0  
community.grafana             1.5.1  
community.hashi_vault         3.1.0  
community.hrobot              1.5.0  
community.libvirt             1.1.0  
community.mongodb             1.4.2  
community.mysql               3.3.0  
community.network             4.0.1  
community.okd                 2.2.0  
community.postgresql          2.2.0  
community.proxysql            1.4.0  
community.rabbitmq            1.2.2  
community.routeros            2.2.0  
community.sap                 1.0.0  
community.sap_libs            1.2.0  
community.skydive             1.0.0  
community.sops                1.2.3  
community.vmware              2.7.0  
community.windows             1.10.0 
community.zabbix              1.7.0  
containers.podman             1.9.4  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.14 
dellemc.enterprise_sonic      1.1.1  
dellemc.openmanage            5.5.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.18.0 
fortinet.fortimanager         2.1.5  
fortinet.fortios              2.1.6  
frr.frr                       2.0.0  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.8.1  
hpe.nimble                    1.1.4  
ibm.qradar                    2.0.0  
ibm.spectrum_virtualize       1.9.0  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.3.0  
inspur.sm                     2.0.0  
junipernetworks.junos         3.1.0  
kubernetes.core               2.3.2  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.18.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.21.0
netapp.storagegrid            21.10.0
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.3.0  
netbox.netbox                 3.7.1  
ngine_io.cloudstack           2.2.4  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.2  
openstack.cloud               1.8.0  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   2.2.0  
purestorage.flasharray        1.13.0 
purestorage.flashblade        1.9.0  
purestorage.fusion            1.0.2  
sensu.sensu_go                1.13.1 
servicenow.servicenow         1.0.6  
splunk.es                     2.0.0  
t_systems_mms.icinga_director 1.30.1 
theforeman.foreman            3.4.0  
vmware.vmware_rest            2.2.0  
vyos.vyos                     3.0.1  
wti.remote                    1.0.4

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.24.50
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/USERNAME/.local/share/virtualenvs/sbsa-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.27.50
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/USERNAME/.local/share/virtualenvs/sbsa-iac-ansible-aws-m1mSr6qK/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
HOST_KEY_CHECKING(/Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg) = False
INTERPRETER_PYTHON(/Users/USERNAME/Development/proj/proj-iac-ansible-aws/ansible.cfg) = /usr/bin/python3.9

OS / Environment

control node: macOS

Steps to Reproduce


---
- name: Provision RDS Cluster and Instances
  hosts: localhost
  connection: local
  gather_facts: False
  vars:
    # SUPPLY VPC INFO HERE
    aws_profile: "{{ lookup('env', 'AWS_PROFILE') | default('default', true) }}"
    db_username: 'my_cluster_username'
    db_password: 'my_cluster_password'
    subnet_group: 'my_cluster_subnet_group'
    parameter_group: 'my_cluster_parameter_group' 
    security_group: 'my_cluster_security_group'
    kms_key_arn: 'my_cluster_kms_key_arn'

  tasks:
    - name: Create Database Cluster
      local_action:
        module: community.aws.rds_cluster
        profile: "{{ aws_profile }}"
        region: "eu-west-1"
        db_cluster_identifier: empty-database-cluster
        engine: "aurora-postgresql"
        engine_version: "11.9"
        username: "{{ db_username }}"
        password: "{{ db_password }}"
        db_subnet_group_name: "{{ subnet_group }}"
        db_cluster_parameter_group_name: "{{ parameter_group }}"
        vpc_security_group_ids: "{{ security_group }}"
        storage_encrypted: yes
        availability_zones:
          - "eu-west-1a"
          - "eu-west-1b"
        kms_key_id: "{{ kms_key_arn }}"
        tags:
          Name: empty-database-cluster
        wait: yes

Expected Results

When you execute the play the first time the cluster is created, when you execute it a second time the playbook fails with an error. I expected it to say 'ok' as in the resource exists and is already created. I

Actual Results

Traceback (most recent call last):
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 107, in <module>
    _ansiballz_main()
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/USERNAME/.ansible/tmp/ansible-tmp-1660299591.748953-13510-114592878762279/AnsiballZ_rds_cluster.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.aws.plugins.modules.rds_cluster', init_globals=dict(_module_fqn='ansible_collections.community.aws.plugins.modules.rds_cluster', _modlib_path=modlib_path),
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1026, in <module>
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 1010, in main
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 872, in ensure_present
  File "/var/folders/5w/y8wg65m578q6y4zdps4w6c6r0000gn/T/ansible_community.aws.rds_cluster_payload_etoe6k1_/ansible_community.aws.rds_cluster_payload.zip/ansible_collections/community/aws/plugins/modules/rds_cluster.py", line 846, in changing_cluster_options
KeyError: 'DBClusterParameterGroupName'

I suspect that there is a difference being detected between what is defined and what has just been created in the environment so the code attempts a method with the wrong attributes.

Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 2 years ago

cc @alinabuzachis @jillr @markuman @s-hertel @tremble click here for bot help

alinabuzachis commented 2 years ago

@imesias Thank you for raising this issue. Will have a look and let you know.

imesias commented 2 years ago

@alinabuzachis @jillr @markuman @s-hertel @tremble I had a minute to have a quick look at this today. It appears the 'KeyError' is valid. get_cluster returns the cluster information returned by the describe_db_clusters boto call, if you have a look at the payload. You'll notice there is no DBClusterParameterGroupName there is however a DBClusterParameterGroup.

I noticed the way you tackled this previously was to pop the values off and compare before it reaches https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/rds_cluster.py#L836 which is where the error occurs because the key DBClusterParameterGroupName doesn't exist in the current_cluster.

I suspect I could add the following to resolve?

desired_db_cluster_parameter_group = modify_params.pop("DBClusterParameterGroupName", None)
if desired_db_cluster_parameter_group:
    if desired_db_cluster_parameter_group != current_cluster["DBClusterParameterGroup"]:
        changing_params["DBClusterParameterGroupName"] = desired_db_cluster_parameter_group

If someone could guide me along the contribution process I'd be happy to submit a PR.

imesias commented 2 years ago

@alinabuzachis @jillr @markuman @s-hertel @tremble PR submitted https://github.com/ansible-collections/community.aws/pull/1417