GCP Dynamic inventory folders

per-lind commented 9 months ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Using the folders variable to the inventory plugin does not work.

AWX version

23.7.0

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[X] Other

Installation method

kubernetes

Modifications

no

Ansible version

core 2.15.9 (AWX EE Latest)

Operating system

No response

Web browser

No response

Steps to reproduce

Create a gcp inventory source. Set source variables to

folders: 
  - folderID
hostnames:
  - name
compose:
  ansible_host: networkInterfaces[0].networkIP

Expected results

All machines from projects in folder added to inventory.

Actual results

ansible-inventory [core 2.15.9]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan  4 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
redirecting (type: inventory) ansible.builtin.gcp_compute to google.cloud.gcp_compute
Loading collection google.cloud from /usr/share/ansible/collections/ansible_collections/google/cloud
Using inventory plugin 'ansible_collections.google.cloud.plugins.inventory.gcp_compute' to process inventory source '/runner/inventory/gcp_compute.yml'
[WARNING]:  * Failed to parse /runner/inventory/gcp_compute.yml with auto
plugin: 'auth_kind'
  File "/usr/local/lib/python3.9/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.9/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/inventory/gcp_compute.py", line 608, in parse
    projects = projects + self.projects_for_folder(config_data, folder)
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/inventory/gcp_compute.py", line 531, in projects_for_folder
    projects_response = self.fetch_projects(config_data, link, query)
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/inventory/gcp_compute.py", line 523, in fetch_projects
    response = auth.get(link, params={"filter": query})
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 85, in get
    return self.full_get(url, **kwargs)
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 150, in full_get
    return self.session().get(url, params=params, **kwargs)
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 194, in session
    self._credentials())
  File "/usr/share/ansible/collections/ansible_collections/google/cloud/plugins/module_utils/gcp_utils.py", line 220, in _credentials
    cred_type = self.module.params['auth_kind']
[WARNING]: Unable to parse /runner/inventory/gcp_compute.yml as an inventory
source
ERROR! No inventory was parsed, please check your configuration and options.

Additional information

Using the same config but filtering on projects works.

projects: 
  - projectID
hostnames:
  - name
compose:
  ansible_host: networkInterfaces[0].networkIP

Using the same config from local machine with ansible and the same service account works.

plugin: google.cloud.gcp_compute
folders:
  - folderID 
service_account_file: credentials.json
auth_kind: serviceaccount
hostnames:
  - name
compose:
  ansible_host: networkInterfaces[0].networkIP

ansible-inventory --graph -i gcp.yml

Can it be that some GCP variable is set in the background that impacts this?

fosterseth commented 8 months ago

is the collection version in the awx execution environment match the one you are running locally? (just to rule out versioning issues)

per-lind commented 8 months ago

Checked and both are running version 1.3.0 of google.cloud (that is the version in awx-ee and I guess that is what is used for the inventory sync?)

tberta commented 3 months ago

I have the same problem here. Inventory doesn't work when using folders. But it works when using projects.

Something very strange is that it works on some folders...

I thought it was a permissions problem on my service account, but eventually I eliminated that hypothesis. With the gcloud commands and the same permissions, the service account can definitely list all tested folders, all projects in folders and all compute instances in projects.

ansible / awx