community.vmware.vmware_vm_inventory not filtering properly with AWX tower

ElioLopez commented 3 years ago

SUMMARY

Hello, I'm fetching our VM from our vcenter using the plugin. However, the filtering is not working properly. The ideal solution will be using tags, but I think that a partially matching name on the VM will suffice. I've configured an inventory named "vmware filter" using a VMware source (INVENTORIES/vmware filter/SOURCES/vmware) in ansible tower

I've followed a documentation from redhat and added this to our source variables:

---
alias_pattern: "{{ config.name }}"
host_filters:  "{{ guest.guestId == 'centos8_64Guest'}}"

and even when our hosts has the guestId field available ( I've listed all the variables fetched for a particular host just for information):

ansible_host: xxxx
config.cpuHotAddEnabled: true
config.cpuHotRemoveEnabled: true
config.guestId: centos8_64Guest
config.hardware.numCPU: 4
config.instanceUuid: 5034881d-fbb6-5bc2-e208-81f4fc3bf1cf
config.name: DEVDOCKER07
config.template: false
config.uuid: 4234d62a-cb11-f2e2-025a-ea73ce5e3bce
guest.guestId: centos8_64Guest
guest.guestState: running
guest.hostName: DEVDOCKER07
guest.ipAddress: xxxx
name: DEVDOCKER07
runtime.maxMemoryUsage: 4096
summary.runtime.powerState: poweredOn

All the VMs (170) are fetched in the inventory. I've filtered just the VMs that have the name "DEVDOCKER" on it using the command line with the following inventory file (credentials removed):

cat inventory.yaml
plugin: community.vmware.vmware_vm_inventory
strict: False
validate_certs: False
with_tags: true
properties:
  - runtime.powerState
  - config.name
  - config.guestId
filters:
  - "'DEVDOCKER' in config.name"
hostnames:
  - config.name
keyed_groups:
  - key: config.guestId
    separator: ''
  - key: tags
    separator: ''

and got the expected answer:

[elio.lopez.admin@admin02 ansible-vmware-inventory]$ ansible-inventory -i vmware.yml --list
{
    "_meta": {
        "hostvars": {
            "DEVDOCKER01": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER01"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER01",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER02": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER02"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER02",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER03": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER03"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER03",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER04": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER04"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER04",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER05": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER05"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER05",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER06": {
                "categories": [],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER06"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER06",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {},
                "tags": []
            },
            "DEVDOCKER07": {
                "categories": [
                    "Operating System"
                ],
                "config": {
                    "guestId": "centos8_64Guest",
                    "name": "DEVDOCKER07"
                },
                "config.guestId": "centos8_64Guest",
                "config.name": "DEVDOCKER07",
                "runtime": {
                    "connectionState": "connected",
                    "powerState": "poweredOn"
                },
                "runtime.connectionState": "connected",
                "runtime.powerState": "poweredOn",
                "tag_category": {
                    "Operating System": [
                        "ansible_managed"
                    ]
                },
                "tag_category.Operating System": [
                    "ansible_managed"
                ],
                "tags": [
                    "ansible_managed"
                ]
            }
        }
    },
    "all": {
        "children": [
            "ansible_managed",
            "centos8_64Guest",
            "ungrouped"
        ]
    },
    "ansible_managed": {
        "hosts": [
            "DEVDOCKER07"
        ]
    },
    "centos8_64Guest": {
        "hosts": [
            "DEVDOCKER01",
            "DEVDOCKER02",
            "DEVDOCKER03",
            "DEVDOCKER04",
            "DEVDOCKER05",
            "DEVDOCKER06",
            "DEVDOCKER07"
        ]
    }
}

to achieve this goal, I've tried the following in source variables:

---
alias_pattern: "{{ config.name }}"
host_filters:  "{{ config.name is match('DEVDOCKER*') }}"

But again all the VMs are fetched. Will be nice to achieve the same thing in tower, but the documented way by redhat doesn't seems to work. I`ve also tried to tweak the "vmware_inventory.ini" mounting it on the ansible-tower-web and ansible-tower-task pods, but it does not seem to have any effect. Could you guy please shade some light on how the filtering should be used? I've opened a ticket with RH but they says that the community plugins are not supported by them. We are using tower on openshift 4.7.

ISSUE TYPE

Bug Report / Documentation

COMPONENT NAME

vmware_vm_inventory plugin

ANSIBLE VERSION

Ansible 2.9.16

OS / ENVIRONMENT

Openshift 4.7.13

EXPECTED RESULTS

That the vmware host are filtered in some way

ACTUAL RESULTS

all the vmware hots are returned

ansibullbot commented 3 years ago

Files identified in the description:

[plugins/inventory/vmware_vm_inventory.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/vmware_vm_inventory.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 3 years ago

cc @Akasurde @Tomorrow9 @goneri @pgbidkar @warthog9 click here for bot help

Akasurde commented 3 years ago

@ElioLopez Thanks for reporting this issue. Could you please update the Ansible Tower version?

Akasurde commented 3 years ago

If I understand your problem correctly, you are able to filter using the command line

ansible-inventory -i vmware.yml --list

But you are not able to filter using Ansible tower with

---
alias_pattern: "{{ config.name }}"
host_filters:  "{{ config.name is match('DEVDOCKER*') }}"

ElioLopez commented 3 years ago

Hello @Akasurde, will try to upgrade the tower and give it another try, but you last message is correct, that's what I'm trying to achieve, maybe I'm not getting the syntax right. Thanks,

ElioLopez commented 3 years ago

Hello @Akasurde, I've upgrade to 3.8.3, but got the same results: the inventory is not properly filtered and all the available hosts on vcenter are retrieved. It's documented on how actually the "source variables" should be used in tower? it doesn't seem to make a difference which value I put on there (used another examples available on the documentation and with same results) Maybe we need to tweak some of the files located on /var/lib/awx/vendor/awx_ansible_collections/ansible_collections/community/vmware/scripts/inventory/. I've collected all of them on this gist for reference, (I'm using the default ones, but maybe are overwritten at some point) Any input will be greatly appreciated. Thanks! Elio

rdzielicki commented 2 years ago

hello, seem having same problem in AWX 14.

JonathanG3 commented 2 years ago

Hey, also in AWX 20.0.0... no filters whatsover, this is my source variables:

properties:
  - "config.name"
  - 'runtime.powerState'
  - "guest.hostName"
  - "guest.ipAddress"
  - "guest.guestState"
  - "guest.guestFamily"
  - "guest.guestId" # Forgot to add this one and it is needed to be able to filter by it
hostnames:
  - config.name
filters:
  - "runtime.powerState == 'poweredOn' and 'windows' in guest.guestId"
compose:
    ansible_host: guest.hostName # need for kerberos authentication
    ansible_connection: "'winrm'"
    ansible_port: 5985
    ansible_winrm_transport: "'kerberos'"
with_tags: true

and I'm still getting even poweredoff machines and linux machines...

EDIT: I managed to get it working by setting the filters: like this:

filters:
- runtime.powerState == "poweredOn" and guest.guestFamily == "windowsGuest"

Another filter option:

filters:
- runtime.powerState == "poweredOn" and (guest.guestId is match('windows.*'))

EDIT 2: I found out that at least in my case, when I'm filtering using a specific property (like 'guest.guestId') I must include it as part of the properties I'm gathering from the vCenter. For example - my correct property list is now this one:

properties:
  - "config.name"
  - 'runtime.powerState'
  - "guest.hostName"
  - "guest.ipAddress"
  - "guest.guestState"
  - "guest.guestFamily"
  - "guest.guestId" # Forgot to add this one and it is needed to be able to filter by it

pidreher commented 1 year ago

@ElioLopez Digging this up. Chances are you don't care about this anymore.

I think you might be confusing the inventory script with the inventory plugin. I am nothing but a humble peasant without RH subscription, therefore I can't read the RedHat issue you've linked. But I believe you refer more or less to this here: https://docs.ansible.com/ansible-tower/3.3.4/html/userguide/inventories.html#vmware-vcenter This is talking about the >inventory script<

In your local example you are using the plugin (plugins/inventory/vmware_vm_inventory.py) In tower you are using the script (scripts/inventory/vmware_inventory.py)

So you are comparing the functionality of two different pieces of code with each other.

I'm not using the script myself and it seems to be too inefficient to run on the quite large vCenter I have on hand for testing. Wild guess would be that I wouldn't be surprised if the host_filters you've defined get mangled by templating before they end up with your script. But I might as well be wrong with that. So I'd try to swap the single and double quotes around.

If you haven't already I'd suggest you use the inventory plugin in tower instead.

pidreher commented 1 year ago

@JonathanG3 it's mentioned in the plugin docs that you need to populate the properties, which you use elsewhere for filtering and groups: Please make sure that all the properties that are used in other parameters are included in this options.

Adding that remark to every possible parameter of the plugin would be cluttering the docs imho. So idk. I guess it's fine the way it is..?

pidreher commented 1 year ago

@Akasurde My guess is that this can be closed.

Maybe somebody wants to test the inventory script in conjunction with tower source_variables and whatnot. I'm not sure if that makes sense, since the script is probably to be considered legacy.

Also generally the initial issue "inventory plugin does not behave the same way in tower as locally" is proven to be a wrong conclusion.

ansible-collections / community.vmware