search

ansible / galaxy_collection

Collection of modules and roles to configure Automation Hub
GNU General Public License v3.0
58 stars 54 forks source link

Lookup plugin ah_api is not working #196

Closed Alex-Bron closed 1 year ago

Alex-Bron commented 1 year ago

Summary

When using the ah_api lookup plugin, tasks fail with message: unknown doc_fragment(s) in file /home/alex/git/aap/collections/ansible_collections/redhat_cop/ah_configuration/plugins/lookup/ah_api.py: infra.ah_configuration.auth_plugin When manually replacing infra.ah_configuration with redhat_cop.ah_configuration, tasksfail with message: An unhandled exception occurred while running the lookup plugin 'redhat_cop.ah_configuration.ah_api'. Error was a <class 'AttributeError'>, original message: 'dict' object has no attribute 'getheaders'. 'dict' object has no attribute 'getheaders'

Issue Type

Ansible, Collection, Private Automation Hub details

ansible --version
ansible [core 2.14.1]
  config file = /home/alex/git/aap/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/alex/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.13 (main, Nov  9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] (/usr/bin/python3.9)
  jinja version = 3.0.3
  libyaml = True

ansible-galaxy collection list
# /usr/share/ansible/collections/ansible_collections
Collection       Version
---------------- -------
redhat.rhel_mgmt 1.1.0  

# /home/alex/.ansible/collections/ansible_collections
Collection              Version
----------------------- -------
ansible.posix           1.4.0  
community.crypto        2.7.1  
community.general       5.8.0  
community.libvirt       1.2.0  
containers.podman       1.9.4  
freeipa.ansible_freeipa 1.8.4  
oasis_roles.system      1.3.0  
redhat.insights         1.0.7  
redhat.rhel_idm         1.8.4

Private Automation Hub version
2.3

OS / ENVIRONMENT

Tasks run on localhost (RHEL8) against Private Automation Hub 2.3 on RHEL 8.7

Desired Behavior

I want to retrieve a list of collections from the rh-certified repository, so I can remove those from the list of requirements.yml entries which then can be configured in the communities repository. The way to retrieve the list of collections is by doing an ah_api lookup for collections on the private automation hub

Actual Behavior

Snippet of code used for this:

- name: Sync the remote collection repositories
  hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: Authenticate and get an API token
      redhat_cop.ah_configuration.ah_token:
        ah_host: "{{ ah_hostname }}"
        ah_username: "{{ ah_username }}"
        ah_password: "{{ ah_password }}"
        validate_certs: "{{ ah_validate_certs | default(true) }}"

    - name: Next, query the Automation Hub for the rh-certified collections
      ansible.builtin.debug:
        msg: "Collections: {{ lookup('redhat_cop.ah_configuration.ah_api', 'collections', host=ah_hostname, oauth_token=ah_token, return_all=true) }}"

    - name: Deauthenticate
      redhat_cop.ah_configuration.ah_token:
        ah_host: "{{ ah_hostname }}"
        ah_token: "{{ ah_token }}"
        validate_certs: "{{ ah_validate_certs | default(true) }}"
        state: absent

Task failure messages: When ah_api is still referring to the infra.ah_configuration collection:

TASK [automation_platform : Next, query the Automation Hub for the rh-certified collections] ***
Sunday 08 January 2023  20:56:02 +0000 (0:00:00.052)       0:00:16.143 ******** 
fatal: [localhost]: FAILED! => {"msg": "unknown doc_fragment(s) in file /home/alex/git/aap/collections/ansible_collections/redhat_cop/ah_configuration/plugins/lookup/ah_api.py: infra.ah_configuration.auth_plugin"}

When ah_api is modified to refer to redhat_cop.ah_configuration:

TASK [automation_platform : Next, query the Automation Hub for the rh-certified collections] ***
Sunday 08 January 2023  20:54:50 +0000 (0:00:00.050)       0:00:16.829 ******** 
fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'redhat_cop.ah_configuration.ah_api'. Error was a <class 'AttributeError'>, original message: 'dict' object has no attribute 'getheaders'. 'dict' object has no attribute 'getheaders'"}

STEPS TO REPRODUCE

Use the code snippet from above to reproduce.

Tompage1994 commented 1 year ago

Hi @Alex-Bron

Thanks for raising this to attention. After checking I can see that ah_token is not actually supported for the ah_api lookup plugin (as the same as several of the modules due to the way it interacts with the API). I will raise a PR to fix the documentation for the plugin but in the meantime the workaround would be to directly apply the username and password to the plugin as follows:

- name: Next, query the Automation Hub for the rh-certified collections
      ansible.builtin.debug:
        msg: "Collections: {{ lookup('redhat_cop.ah_configuration.ah_api', 'collections', host=ah_hostname, username=ah_username, password=ah_password, return_all=true) }}"
Tompage1994 commented 1 year ago

Additionally, for the doc_fragment issue, there is not a great workaround for this as is a symptom of us moving the namespace of this collection. My advice would be to make use of the new location for the collection: infra.ah_configuration as the redhat_cop namespace will eventually be deprecated for this collection.

Alex-Bron commented 1 year ago

Thanks for the very swift replies to my issue. So I changed the namespace to infra (will redhat_cop.controller_configuration also be replaced by infra.controller_configuration?) and added my username and password to the ah_api lookup, however this returns an empty list although I'm sure there are collections present on my automation hub. Am I maybe doing something wrong here? Is my assumption incorrect that this ah_api lookup can be used to retrieve a list of collections present on the system? In the end I'd want to query for a list of collections from the rh-certified repository, as those should not be searched for in the community repository.

Code snippet:

- name: Next, query the Automation Hub for the rh-certified collections
  ansible.builtin.debug:
    msg: "Collections: {{ lookup('infra.ah_configuration.ah_api', 'collections', host=ah_hostname, username=ah_username, password=ah_password, return_all=true) }}"
Tompage1994 commented 1 year ago

Ah yes, looks like the API for this has changed. It seems this will only be pulling the published collections now. I have raised an issue which will allow an additional term to be given which pulls from a specific repository (e.g. rh-certified). I'll write a PR up for this and we can try to get it in ASAP. Unfortunately for now, there is no way to query the list of collections under anything other than published

Alex-Bron commented 1 year ago

Ok, waiting for the 1.0.1 release then :-) I hope my approach makes sense: query all gitlab projects for their requirements.yml, carve out the rh-certified collections from that list, and add the remainder to the communities repository for syncing (I don't use "my own" collections so no reason to carve out also the published ones...)

Tompage1994 commented 1 year ago

That's a pretty neat solution to automatically work out what you need to sync, so definitely this update will help you out.

Tompage1994 commented 1 year ago

@Alex-Bron v1.0.1 has now been released here: https://galaxy.ansible.com/infra/ah_configuration

Hope this is now all fixed for you

Alex-Bron commented 1 year ago

Thanks Tom for the effort, downloaded the new collection and tested it and yes! I can get a json with all rh-certified collections. Will continue my quest for generating the community requirements from this (I really want to automate the automation as much as possible :-) )