search

ansible-collections / amazon.aws

Ansible Collection for Amazon AWS
GNU General Public License v3.0
304 stars 334 forks source link

Cannot use launch_template name in ec2_instance module #451

Closed markuman closed 2 years ago

markuman commented 3 years ago

Two issues (TL;DR)

  1. Cannot use launch_template name in ec2_instance module
    • broken condition
  2. You cannot use launch_template paremeter with other ec2_instance parameters in the same task (e.g. instance_type).
    • that's a general boto3 limitation
    • default values of ec2_instance is making this even more diffcult to handle

Summary

Origin Issue: https://github.com/ansible/ansible/issues/51941
First trial of fix: https://github.com/ansible-collections/community.aws/pull/111
Broken condition still exists: https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/ec2_instance.py#L1165

However. It's not that easy.

Imo, LaunchTemplate parameters are not changeable in the context of run_instance.
You must read-out the launch_template parameters and throw them aginst run_instance to pretend they came from module.params.

"Failed to create new EC2 instance: An error occurred (Unsupported) when calling the RunInstances operation: The requested configuration is currently not supported. Please check the documentation for supported configurations."

One issues are the default values the modules applies.
{'BlockDeviceMappings': [], 'ClientToken': '37d405af374948e2bd427ff4a7a7fc04', 'InstanceType': 't2.micro', 'LaunchTemplate': {'LaunchTemplateName': 'ubuntu-20.04'}, 'MaxCount': 1, 'MinCount': 1, 'NetworkInterfaces': [{'DeviceIndex': 0, 'SubnetId': 'subnet-3e5fe743'}], 'TagSpecifications': [{'ResourceType': 'volume', 'Tags': [{'Key': 'Name', 'Value': 'lttest01'}]}, {'ResourceType': 'instance', 'Tags': [{'Key': 'Name', 'Value': 'lttest01'}]}]}

This happen when you run ec2_instance without any parameters, just with a LaunchTemplateName. So the module applies

basically when I run 

 - name: test launchtemplate usage
      ec2_instance:
        wait: yes
        name: "{{ name }}"
        instance_type: c5a.large
        launch_template:
          name: ubuntu-20.04
        vpc_subnet_id: subnet-3e5fe743

I expect that all values are taken from the launch_template, only the requested will be overwritten (instance_type and vpc_subnet_id.)

but because of the default values, when I request

 - name: test launchtemplate usage
      ec2_instance:
        wait: yes
        name: "{{ name }}"
        launch_template:
          name: ubuntu-20.04
        vpc_subnet_id: subnet-3e5fe743

what instance_type will be choosen? I expect the one that is defined in the launch_template (e.g. t3a.small). But the module will request a t2.micro instance (default value in ec2_instance).

so one point that must be reworked are the parameters that breaks the launch_template when no extra parameters are requested.
that might be the easiert part.

@@ -1273,16 +1262,17 @@ def build_run_instance_spec(params, ec2=None):
         MinCount=1,
     )
     # network parameters
-    spec['NetworkInterfaces'] = build_network_spec(params, ec2)
-    spec['BlockDeviceMappings'] = build_volume_spec(params)
-    spec.update(**build_top_level_options(params))
-    spec['TagSpecifications'] = build_instance_tags(params)
+    if not params.get('launch_template'):
+        spec['NetworkInterfaces'] = build_network_spec(params, ec2)
+        spec['BlockDeviceMappings'] = build_volume_spec(params)
+        spec['TagSpecifications'] = build_instance_tags(params)
+        # IAM profile
+        if params.get('instance_role'):
+            spec['IamInstanceProfile'] = dict(Arn=determine_iam_role(params.get('instance_role')))
+        spec['InstanceType'] = params['instance_type']

-    # IAM profile
-    if params.get('instance_role'):
-        spec['IamInstanceProfile'] = dict(Arn=determine_iam_role(params.get('instance_role')))
+    spec.update(**build_top_level_options(params))

-    spec['InstanceType'] = params['instance_type']
     return spec

the harder part is to read-out launch_template parameters and compare them to ec2_instance parameters if requested and throw everything against run_instance without LaunchTemplate parameter.

Issue Type

Bug Report

Component Name

ec2_instance

Ansible Version

ansible [core 2.11.3] 
  config file = None
  configured module search path = ['/home/m/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/m/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/m/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/m/.local/bin/ansible
  python version = 3.9.6 (default, Jun 30 2021, 10:22:16) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True

Collection Versions

ansible-galaxy collection list

# /home/m/.ansible/collections/ansible_collections
Collection           Version  
-------------------- ---------
amazon.aws           1.5.0    
ansible.netcommon    1.5.0    
community.aws        1.5.0    
community.crypto     1.6.1    
community.dns        2.0.0-a1 
community.docker     1.4.0    
community.general    3.0.0    
community.libvirt    1.0.1    
community.mysql      2.1.0    
community.proxysql   1.2.0-dev
devsec.hardening     7.5.0    
google.cloud         1.0.1    
hetzner.hcloud       1.4.4    
lekker.aws           0.0.3    
markuman.hetzner_dns 1.6.1    
markuman.nextcloud   9.0.0    

# /home/m/.local/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.0  
ansible.netcommon             2.2.0  
ansible.posix                 1.2.0  
ansible.utils                 2.3.0  
ansible.windows               1.7.0  
arista.eos                    2.2.0  
awx.awx                       19.2.2 
azure.azcollection            1.7.0  
check_point.mgmt              2.0.0  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.0.0  
cisco.asa                     2.0.2  
cisco.intersight              1.0.15 
cisco.ios                     2.3.0  
cisco.iosxr                   2.3.0  
cisco.meraki                  2.4.2  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.4.0  
cisco.ucs                     1.6.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 1.5.0  
community.azure               1.0.0  
community.crypto              1.7.1  
community.digitalocean        1.8.0  
community.docker              1.8.0  
community.fortios             1.0.0  
community.general             3.4.0  
community.google              1.0.0  
community.grafana             1.2.1  
community.hashi_vault         1.3.2  
community.hrobot              1.1.1  
community.kubernetes          1.2.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.1  
community.mongodb             1.2.1  
community.mysql               2.1.0  
community.network             3.0.0  
community.okd                 1.1.2  
community.postgresql          1.4.0  
community.proxysql            1.0.0  
community.rabbitmq            1.0.3  
community.routeros            1.2.0  
community.skydive             1.0.0  
community.sops                1.1.0  
community.vmware              1.12.0 
community.windows             1.5.0  
community.zabbix              1.4.0  
containers.podman             1.6.1  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.7  
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            3.5.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.10.1 
fortinet.fortimanager         2.1.3  
fortinet.fortios              2.1.2  
frr.frr                       1.0.3  
gluster.gluster               1.0.1  
google.cloud                  1.0.2  
hetzner.hcloud                1.4.4  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.2.4  
inspur.sm                     1.2.0  
junipernetworks.junos         2.3.0  
kubernetes.core               1.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.6.0 
netapp.azure                  21.8.1 
netapp.cloudmanager           21.8.0 
netapp.elementsw              21.6.1 
netapp.ontap                  21.8.1 
netapp.um_info                21.7.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.1.1  
ngine_io.cloudstack           2.1.0  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.0  
openvswitch.openvswitch       2.0.0  
ovirt.ovirt                   1.5.3  
purestorage.flasharray        1.9.0  
purestorage.flashblade        1.6.0  
sensu.sensu_go                1.11.1 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.20.0 
theforeman.foreman            2.1.2  
vyos.vyos                     2.4.0  
wti.remote                    1.0.1

AWS SDK versions

pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /home/m/.local/lib/python3.9/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.18.4
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /home/m/.local/lib/python3.9/site-packages
Requires: s3transfer, jmespath, botocore
Required-by: awslogs
---
Name: botocore
Version: 1.21.4
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /home/m/.local/lib/python3.9/site-packages
Requires: jmespath, urllib3, python-dateutil
Required-by: s3transfer, boto3, awscli

Configuration

OS / Environment

Linux rocket 5.11.22-2-MANJARO #1 SMP PREEMPT Fri May 21 17:45:54 UTC 2021 x86_64 GNU/Linux

Steps to Reproduce


    - name: find ubuntu ami
      ec2_ami_info:
        owners: 099720109477
        region: eu-central-1
        filters:
          name: ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20201026
      register: ami

    - name: find default security group
      amazon.aws.ec2_group_info:
        region: eu-central-1
        filters:
          group-name:
            - default
      register: SG

    - name: ubuntu 20.04 launch template
      community.aws.ec2_launch_template:
        template_name: ubuntu-20.04
        image_id: "{{ ami.images[0].image_id }}"
        instance_type: t3a.small
        credit_specification:
          cpu_credits: unlimited
        network_interfaces:
          - associate_public_ip_address: no
            delete_on_termination: yes
            device_index: 0
            groups:
              - "{{ SG.security_groups[0].group_id }}"
            subnet_id:  "{{ subnet }} "
        user_data: "{{ "uname -a" | b64encode }}"
        region: "{{ region }}"
        key_name: "{{ keypair }}"
        ebs_optimized: yes
        block_device_mappings:
          - device_name: /dev/sda1
            ebs:
              volume_size: 8
              volume_type: gp3
              delete_on_termination: true
              kms_key_id: "{{ KMS_ARN }}"
              encrypted: yes
          - device_name: /dev/sdb
            ebs:
              volume_size: 5
              volume_type: gp3
              delete_on_termination: true
              kms_key_id: "{{ KMS_ARN }}"
              encrypted: yes
        tags:
          ssome: tags

     - name: test launchtemplate usage
      ec2_instance:
        wait: yes
        name: "{{ name }}"
        instance_type: c5a.large
        launch_template:
          name: ubuntu-20.04
        vpc_subnet_id: subnet-3e5fe743

Expected Results

Deploy an ec2 instance with lauch_template parameters for all parameters that are not requested by ec2_instance module.

Actual Results

TASK [Provison ec2 instance(s)] ***********************************************************************************************
failed: [localhost] (item=boom) => {"changed": false, "item": "boom", "msg": "Could not create instance with launch template. Either launch_template.name or launch_template.id parameters are required"}

Code of Conduct

ansibullbot commented 3 years ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 3 years ago

cc @jillr @ryansb @s-hertel @tremble @wimnat click here for bot help

markuman commented 3 years ago

maybe the implementation is not that difficult at the end.
the only question that is left: what to do with the default values?

One possibility is, that they are only applied when launch_template parameter is not requested. 

if not module.params.get('launch_template'):
    instance_type = 't2.micro'

But that means that they must remove from AnsibleAWSModule() initialisation, because you cannot differentiate if module.params.get('instance_type') is a requested value or a default value. Or is it possible?
Removing default values from AnsibleAWSModule() will result in doc errors I think.

Removing the default value in general will be the easiest fix. But that must be made in some iteration steps with deprecation warnings?

aaron-p-lehmann commented 2 years ago

I would love to see this implemented. IMO, the best behavior would be to apply the values in the following order:

I have 0 knowledge of boto, so I don't know how easy that is to make happen, though.

markuman commented 2 years ago

I think its vice-versa :)

  1. take parameter values that are set by user in ec2_instance task.
  2. take default values for all parameters that a not defined and have a default value.

If launch template is defined

  1. take launch_template parameter values
  2. overwrite launch_template parameter values that are set by user in ec2_instance task.

I have 0 knowledge of boto, so I don't know how easy that is to make happen, though.

I think it's not that difficult, but a lot of time consuming (adjust and expand integration tests).
If you like to get involved, you just missed the ansible-fest 2022. But there are many helping hands on libera irc in #ansible-aws and also #ansible-community

aaron-p-lehmann commented 2 years ago

I was thinking of the numbers from least important to most, and you were going vice versa, I think. In any case, I agree with you. I'd be interested in getting involved, but I don't know how much time I'd have to maintain stuff. Also, at my work, we're tied to ansible 2.9, unfortunately, so I wouldn't be able to use and test the thing very much. I'll definitley look into it, though, if we ever get things sorted out so that we can upgrade our ansible.