search

ansible-collections / amazon.aws

Ansible Collection for Amazon AWS
GNU General Public License v3.0
304 stars 332 forks source link

RDS - rds_instance - performance_insights_kms_key_id #2217

Closed eRadical closed 3 weeks ago

eRadical commented 1 month ago

Summary

When creating a new rds_instance with a custom "performance_insights_kms_key_id" (not the default one "aws/rds") the KMS key is not applied but the default one is.

Issue Type

Bug Report

Component Name

rds_instance

Ansible Version

$ ansible --version
ansible [core 2.17.2]
  config file = None
  configured module search path = ['/home/.../.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/.../.local/lib/python3.12/site-packages/ansible
  ansible collection location = /home/.../.ansible/collections:/usr/share/ansible/collections
  executable location = /home/.../.local/bin/ansible
  python version = 3.12.4 (main, Jun  7 2024, 00:00:00) [GCC 14.1.1 20240607 (Red Hat 14.1.1-5)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /home/.../.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               8.1.0  
ansible.posix                            1.5.4  
community.aws                            8.0.0  
community.general                        8.6.0  
community.mongodb                        1.7.4  
community.mysql                          3.9.0  
kubernetes.core                          3.1.0

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.33.11
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /home/.../.local/lib/python3.12/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.33.11
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /home/.../.local/lib/python3.12/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CACHE_PLUGIN(/home/.../ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/.../ansible.cfg) = ~/.ansible/cache
CACHE_PLUGIN_TIMEOUT(/home/.../ansible.cfg) = 3600
CALLBACKS_ENABLED(/home/.../ansible.cfg) = ['timer', 'profile_tasks', 'profile_roles']
CONFIG_FILE() = /home/.../ansible.cfg
DEFAULT_ASK_PASS(/home/.../ansible.cfg) = False
DEFAULT_EXECUTABLE(/home/.../ansible.cfg) = /bin/bash
DEFAULT_FORCE_HANDLERS(/home/.../ansible.cfg) = True
DEFAULT_FORKS(/home/.../ansible.cfg) = 15
DEFAULT_HOST_LIST(/home/.../ansible.cfg) = ['/home/.../envs']
DEFAULT_LOG_PATH(/home/.../ansible.cfg) = /home/.../.ansible/ansible.log
DEFAULT_MANAGED_STR(/home/.../ansible.cfg) = Ansible managed! DON'T CHANGE THIS FILE BY HAND! You were warned!
DEFAULT_ROLES_PATH(/home/.../ansible.cfg) = ['/home/.../roles']
DEFAULT_TIMEOUT(/home/.../ansible.cfg) = 30
DEPRECATION_WARNINGS(/home/.../ansible.cfg) = True
EDITOR(env: EDITOR) = /usr/bin/vim
HOST_KEY_CHECKING(/home/.../ansible.cfg) = False
INVENTORY_ENABLED(/home/.../ansible.cfg) = ['yaml', 'amazon.aws.aws_ec2', 'amazon.aws.aws_rds', 'ini']
RETRY_FILES_ENABLED(/home/.../ansible.cfg) = False
SHOW_CUSTOM_STATS(/home/.../ansible.cfg) = True
TASK_TIMEOUT(/home/.../ansible.cfg) = 7200

OS / Environment

Fedora release 40 (Forty)

Steps to Reproduce

- name: Ensure RDS «PRIMARY» Instances are present
  amazon.aws.rds_instance:
    db_instance_identifier: "{{ item.name | mandatory }}"
    state: "{{ item.state | default('present', true) }}"
    engine: "{{ item.engine | mandatory }}"
    engine_version: "{{ item.engine_version | mandatory }}"
    ...
    enable_performance_insights: "{{ item.enable_performance_insights | default(true) }}"
    performance_insights_retention_period: "{{ item.performance_insights_days | default(our_performance_insights_days) }}"
#    performance_insights_kms_key_id: "{{ our_aws_kms_ids[item.instance_kms_key_id] }}"
#    performance_insights_kms_key_id: "arn:aws:kms:eu-west-1:565656565656:key/mrk-b9a9d5be4a8ade74957e62af13954599"
    performance_insights_kms_key_id: "mrk-b9a9d5be4a8ade74957e62af13954599"
    ...
    monitoring_interval: 60
    kms_key_id: "{{ our_aws_kms_ids[item.instance_kms_key_id] }}"
    # boto stuff & tags
    profile: "{{ aws_profile }}"
    region: "{{ item.region | default(our_account_default_region) }}"

Expected Results

I expect to see the key declared not the default "aws/rds"

Actual Results

in response we have:

"performance_insights_kms_key_id": "arn:aws:kms:eu-west-1:565656565656:key/4063-870a-6da5c69cedff-6f2d4e3e-0635",

Code of Conduct

eRadical commented 1 month ago

I looked in CloudTrail on the CreateDBInstance instance and I found out that AWS does not see Ansible sending a different performance_insights_kms_key_id.

Actually in the event record in "requestParameters" I do not have at all "performanceInsightsKMSKeyId". I do see "kmsKeyId" set to the same key.

In the "responseElements" I can see the "kmsKeyId" applied correctly and I also can see the wrong

"performanceInsightsKMSKeyId": "arn:aws:kms:eu-west-1:565656565656:key/4063-870a-6da5c69cedff-6f2d4e3e-0635",

which is actually the default "aws/rds".

eRadical commented 1 month ago

I suspect that the issue is with capitalization of parameters:

Note that in the second KMS is all caps.

I have tried to create an instance via AWS console and it works ok - I get the custom key in performance insights. I'll try tomorrow via aws cli to see if this is a boto issue.

eRadical commented 1 month ago

I'm willing to debug this if anyone can hint me on how to get started.

eRadical commented 1 month ago 
aws rds create-db-instance --db-instance-identifier=db-prod-59 \
    --db-instance-class=db.m6i.large --engine=mariadb \
    --enable-performance-insights \
    --performance-insights-kms-key-id="mrk-b9a9d5be4a8ade74957e62af13954599" \
   ... --storage-encrypted --kms-key-id="mrk-b9a9d5be4a8ade74957e62af13954599"

I just tried w/ the above and it creates the DB w/ correct keys. So 99% the problem is in Ansible -> rds_instance.

eRadical commented 1 month ago

I might have a confirmation that it is an issue with capitalization of parameters.

In line: https://github.com/ansible-collections/amazon.aws/blob/b63f26dcb28a120c5d00210a5d59adc7b906fb82/plugins/module_utils/rds.py#L580

I added KMS:

for old, new in (("Db", "DB"), ("Iam", "IAM"), ("Az", "AZ"), ("Ca", "CA"), ("Kms", "KMS")):

But this time the DB instance was created w/ the correct key for performance-insights but w/ the wrong key for kms_key_id (it used the default "aws/rds").