ansible-collections / amazon.aws

Ansible Collection for Amazon AWS
GNU General Public License v3.0
304 stars 334 forks source link

ec2 inventory lost instances if customize dns is set via dhcp options #710

Closed link89 closed 2 years ago

link89 commented 2 years ago

Summary

When I use the default dhpc that assign to my VPC, and use aws_ec2 inventory, everything works fine. But after I create a customize dhcp options that use dns servers (like 8.8.8.8)and set my VPC to use it, the output of the same inventory script will become empty.

Issue Type

Bug Report

Component Name

aws_ec2 inventory

Ansible Version

$ ansible --version
ansible [core 2.12.3]
  config file = None
  configured module search path = ['/Users/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
  ansible collection location = /Users/xxxx/.ansible/collections:/usr/share/ansible/collections
  executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible
  python version = 3.8.7 (v3.8.7:6503f05dd5, Dec 21 2020, 12:45:15) [Clang 6.0 (clang-600.0.57)]
  jinja version = 3.0.1
  libyaml = False

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0
ansible.netcommon             2.5.1
ansible.posix                 1.3.0
ansible.utils                 2.5.0
ansible.windows               1.9.0
arista.eos                    3.1.0
awx.awx                       19.4.0
azure.azcollection            1.11.0
check_point.mgmt              2.2.2
chocolatey.chocolatey         1.2.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.18
cisco.ios                     2.7.1
cisco.iosxr                   2.7.0
cisco.ise                     1.2.1
cisco.meraki                  2.6.0
cisco.mso                     1.3.0
cisco.nso                     1.0.3
cisco.nxos                    2.9.0
cisco.ucs                     1.6.0
cloud.common                  2.1.0
cloudscale_ch.cloud           2.2.0
community.aws                 2.3.0
community.azure               1.1.0
community.ciscosmb            1.0.4
community.crypto              2.2.2
community.digitalocean        1.15.1
community.dns                 2.0.7
community.docker              2.2.0
community.fortios             1.0.0
community.general             4.5.0
community.google              1.0.0
community.grafana             1.3.2
community.hashi_vault         2.3.0
community.hrobot              1.2.2
community.kubernetes          2.0.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.4
community.network             3.0.0
community.okd                 2.1.0
community.postgresql          1.7.0
community.proxysql            1.3.1
community.rabbitmq            1.1.0
community.routeros            2.0.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.17.1
community.windows             1.9.0
community.zabbix              1.5.1
containers.podman             1.9.1
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            4.4.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.14.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.4
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.4
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.3
infoblox.nios_modules         1.2.1
inspur.sm                     1.3.0
junipernetworks.junos         2.9.0
kubernetes.core               2.2.3
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.14.0
netapp.elementsw              21.7.0
netapp.ontap                  21.16.0
netapp.storagegrid            21.9.0
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.5.1
ngine_io.cloudstack           2.2.3
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.7.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   1.6.6
purestorage.flasharray        1.12.1
purestorage.flashblade        1.9.0
sensu.sensu_go                1.13.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.27.1
theforeman.foreman            2.2.0
vyos.vyos                     2.7.0
wti.remote                    1.0.3

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.21.19
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.24.19
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

MacOS, CentOS

Steps to Reproduce

plugin: aws_ec2
regions:
  - eu-central-1
keyed_groups:
 - key: tags
   prefix: tag
 - prefix: instance_type
   key: instance_type
 - key: placement.region
   prefix: aws_region
compose:
  ansible_host: private_ip_address

And run with ansible-inventory -vvvv --graph -i ansible/aws_ec2.yml

Expected Results

List all instances.

Actual Results

@all:
plugin: aws_ec2
  |--@aws_ec2:
  |--@ungrouped:

Code of Conduct

ansibullbot commented 2 years ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

abikouo commented 2 years ago

@link89 Thanks for reporting this issue. I tried to reproduce using this playbook following all steps provided above, but could not

- hosts: localhost
  gather_facts: no

  vars:
    ec2_ami_name: 'Fedora-Cloud-Base-*.x86_64*'

  module_defaults:
    group/aws:
      aws_access_key: "{{ aws_access_key }}"
      aws_secret_key: "{{ aws_secret_key }}"
      security_token: "{{ security_token | default(omit) }}"
      region: "{{ aws_region }}"

  collections:
    - community.aws
    - amazon.aws

  tasks:
    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

    - name: get image ID to create an instance
      ec2_ami_info:
        filters:
          architecture: x86_64
          owner-id: '{{ aws_owner_id }}'
          virtualization-type: hvm
          root-device-type: ebs
          name: 'Fedora-Atomic-27*'
      register: _images
      no_log: true

    - set_fact:
        image_id: '{{ _images.images.0.image_id }}'
        vpc_cidr: '10.{{ 256 | random(seed=resource_prefix) }}.0.0/16'
        subnet_cidr: '10.{{ 256 | random(seed=resource_prefix) }}.0.0/24'

    - name: create a VPC to work in
      ec2_vpc_net:
        cidr_block: '{{ vpc_cidr }}'
        name: '{{ resource_prefix }}-vpc'
        resource_tags:
          Name: '{{ resource_prefix }}'
      register: _vpc

    - set_fact:
        vpc_id: '{{ _vpc.vpc.id }}'

    - name: create a subnet to use for creating an ec2 instance
      ec2_vpc_subnet:
        vpc_id: '{{ vpc_id }}'
        cidr: '{{ subnet_cidr }}'
        resource_tags:
          Name: '{{ resource_prefix }}'
      register: _subnet

    - set_fact:
        subnet_id: '{{ _subnet.subnet.id }}'

    - name: create a security group to use for creating an ec2 instance
      ec2_group:
        name: '{{ resource_prefix }}-sg'
        vpc_id: '{{ vpc_id }}'
        description: "create security group for testing"
      register: _secgroup

    - name: create a new host
      ec2:
        image: '{{ image_id }}'
        count_tag:
          Name: '{{ resource_prefix }}'
        instance_tags:
          Name: '{{ resource_prefix }}'
        instance_type: t2.micro
        wait: yes
        group_id: '{{ _secgroup.group_id }}'
        vpc_subnet_id: '{{ subnet_id }}'

    - meta: refresh_inventory

    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

    - name: Create custom DHCP options
      ec2_vpc_dhcp_option:
        region: "{{ aws_region }}"
        dns_servers:
          - 8.8.8.8
        tags:
          Name: custom servers
          Environment: Test
      register: _dhcpoptions

    - name: update VPC with DHCP options
      ec2_vpc_net:
        cidr_block: '{{ vpc_cidr }}'
        name: '{{ resource_prefix }}-vpc'
        resource_tags:
          Name: '{{ resource_prefix }}'
        dhcp_opts_id: "{{ _dhcpoptions.dhcp_options_id }}"
      register: _vpc

    - meta: refresh_inventory

    - debug:
        msg: "ec2 hosts: {{ groups.aws_ec2 }}"

Here after the output

PLAY [localhost] ************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: []"
}

TASK [get image ID to create an instance] ***********************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a VPC to work in] **********************************************************************************************************************************************
changed: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a subnet to use for creating an ec2 instance] ******************************************************************************************************************
changed: [localhost]

TASK [set_fact] *************************************************************************************************************************************************************
ok: [localhost]

TASK [create a security group to use for creating an ec2 instance] **********************************************************************************************************
changed: [localhost]

TASK [create a new host] ****************************************************************************************************************************************************
changed: [localhost]

TASK [meta] *****************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: ['ip-10-3-0-182.ec2.internal']"
}

TASK [Create custom DHCP options] *******************************************************************************************************************************************
ok: [localhost]

TASK [update VPC with DHCP options] *****************************************************************************************************************************************
changed: [localhost]

TASK [meta] *****************************************************************************************************************************************************************

TASK [debug] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "ec2 hosts: ['ip-10-3-0-182.ec2.internal']"
}

Could you please provide a playbook to reproduce this issue ?

alinabuzachis commented 2 years ago

@link89 Could you please let us know if the issues still persist and in case have a look at what Aubin suggested?

ansibullbot commented 2 years ago

@link89 This issue is waiting for your response. Please respond or the issue will be closed.

click here for bot help

link89 commented 2 years ago

Hi @abikouo @alinabuzachis We figure out a fix, I think we are missing hostnames sections in our setup.

hostnames:
 - private-ip-address

I don't know why it works if we don't change the dhcp setting thought. But it works fine now.