AWX - EC2 dynamic inventory source not grouping properly

ivantomica commented 2 years ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I am not entitled to status updates or other assurances.

Summary

Setup I'm working with consists of AWX and a single inventory which has multiple sources consisting of 1 static (sourced from a project) and around 30 dynamic (Amazon EC2) inventory types.

Configuration of each dynamic inventory source is done by using following configuration:

---
# Set Inventory hostname
hostnames:
  - "tag:Name"
  - dns-name
# Add hosts only tagged with Ansible tag
filters:
  tag:Ansible:
    - "True"
    - "true"
  instance-state-name: running
keyed_groups:
  # Add hosts to tag_Name_Value groups for each Name/Value tag pair
  - prefix: tag
    key: tags
  # Group hosts per AWS Owner ID
  - prefix: owner
    key: owner_id | string
compose:
  ansible_host: public_ip_address
  ansible_port: tags.SSHPort|default("22")

Usually, this configuration works as it should from my perspective, as I get the inventory consisting of:

only hosts that are running
only hosts that have tag "Ansible=True"
all hosts are grouped per tags and per owner

To have the inventory up-to-date, there are scheduled jobs which trigger during the night and around 3-4-5 inventories get sourced at the time with 15 minute pause in between batches.

Issue I'm experiencing with automatic sync is that not all hosts are grouped under the tag_Ansible_True group, but the behavior is hit'n'miss. Sometimes all hosts are under the tag_Ansible_True group, but most of the time there are only about 40-60 hosts there while there should be much more. When manual sync is trigger by going to Inventories > NAME_OF_THE_INVENTORY -> Sources -> Sync all, all hosts show up under the tag_Ansible_True.

While in "broken state", examining any of the hosts in the inventory that's missing from the tag_Ansible_True group I can see that its tags are properly populated

tags:
  Ansible: 'True'
  Environment: staging
  Name: somehostname

This points me into direction that something happens during the host grouping phase while they're being grouped and saved into the inventory. Unfortunately, I'm not so familiar with that part of the system and would appreciate further guidance on how to troubleshoot this further.

Multiple curious things happen:

sometimes all works, sometimes doesn't
manual sync-all so far always fixed the issue - without any configuration changes
all hosts have Ansible=True tag (even those missing from the group)
usually - whole AWS Account/Inventory source is missing from the group
hosts missing tag_Ansible_True are also missing aws_ec2 group

AWX version

19.5.0

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Dynamic inventory of ~30 soures - Amazon EC2 source - Each source on different AWS account
Configuration set up as in description example
Scheduled sync doesn't import all the hosts all the time.

Expected results

All hosts that have tag Ansible=True are grouped into tag_Ansible_True group

Actual results

Only a subset of hosts that have tag Ansible=True is in tag_Ansible_True Group

Additional information

No response

pratikbin commented 2 years ago

Hey @ivantomica I think I'm relating to this issue.

Configuration of each dynamic inventory source is done by using following configuration:

Can you point me where we can add this configuration, Because I can't

ivantomica commented 2 years ago

Can you point me where we can add this configuration, Because I can't find where to put this configuration?

In AWX you create dynamic inventory source by going to

Inventories
INVENTORYNAME
Sources
Add
Source: Amazon EC2
Source variables field

This source can then be scheduled via schedules tab.

pratikbin commented 2 years ago

So I have to put this 👇 configuration in Source variables field right!

---
# Set Inventory hostname
hostnames:
  - "tag:Name"
  - dns-name
...

pratikbin commented 2 years ago

Thanks @ivantomica , I was confused with the title itself Source variables.

I wanted to get all the nodes with specific tag. I used Enabled Variable and Enabled value but it disabled that particular node in inventory and enable rest. It is kind of inverse that i thought. But this (Source variables) helped thanks a lot

ivantomica commented 2 years ago

Has anyone experienced similar issues or have similar use-case as we do? Upgrades to the AWX haven't yet resolved the issue, and I am having hard time pinpointing it.

Any guidance on how to troubleshoot this further would be greatly appreciated.

pratikbin commented 2 years ago

I'm not facing this issue in awx 0.21.0

ansible / awx