Cant sync ovh inventory on AWX 16.0.0 and 17.0.0 with region name

bdoublet91 commented 3 years ago

ISSUE TYPE

Bug Report

SUMMARY

Try to configure openstack synchronisation with new AWX version 17.0.0. My region is GRA7 but only download servers from UK1 like version and get error 401 in logs Same behavior with version 16.0.0

ENVIRONMENT

AWX version: 17.0.0
AWX install method: docker on linux
Ansible version: 2.9.16
Operating System: Ubuntu (Awx official docker image)
Web Browser: Firefox

STEPS TO REPRODUCE

Create openstack credential and specify region GRA7 Username and password are the same when I connect through horizon portal, I have also application Key with secret and consumer key. Tried all

EXPECTED RESULTS

Get 20 servers from GRA7 region

ACTUAL RESULTS

Get 0 server from UK1 region in logs but tested with one server on UK1 and it worked

ADDITIONAL INFORMATION

Create a server in UK1 region and setup credential without region name 20 servers in GRA7region

First test: No region name Curl all regions and get 1 server from UK1 region and 0 server from GRA7 Get error 401 and warning

REQ: curl -g -i -X GET https://compute.uk1.cloud.ovh.net/v2.1 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.37.0 keystoneauth1/3.18.0 python-requests/2.22.0 CPython/3.6.8"
RESP: [401] Content-Length: 114 Content-Type: application/json Date: Tue, 26 Jan 2021 13:29:13 GMT Strict-Transport-Security: max-age=15768000 Www-Authenticate: Keystone uri='https://auth.cloud.ovh.net:443/' X-Compute-Request-Id: req-7c7adec3-fea9-4c0c-aa1d-ffb8b31e4247 X-IPLB-Instance: 24841
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Parsed /tmp/awx_25_so4g1bx_/openstack.yml inventory source with auto plugin
    5.714 INFO     Processing JSON output...
    5.716 DEBUG    Loaded group: all
    5.716 DEBUG    Loaded group: _nova
    5.719 DEBUG    Adding child group _nova to parent all
    5.719 DEBUG    Loaded host: s1-2-uk1
    5.719 DEBUG    Adding host s1-2-uk1 to group _nova
    5.720 DEBUG    Adding child group _nova to parent all
    5.720 DEBUG    Adding child group _nova to parent all
    5.720 DEBUG    Loaded group: devstack
    5.721 DEBUG    Adding child group devstack to parent all
    5.721 DEBUG    Loaded group: devstack_
    5.721 DEBUG    Adding child group devstack_ to parent all
    5.722 DEBUG    Loaded group: devstack__nova
    5.722 DEBUG    Adding child group devstack__nova to parent all
    5.722 DEBUG    Loaded group: instance-***
    5.722 DEBUG    Adding child group instance-**** to parent all
    5.723 DEBUG    Loaded group: nova
    5.723 DEBUG    Adding child group nova to parent all
    5.723 DEBUG    Adding host s1-2-uk1 to group devstack
    5.724 DEBUG    Adding child group devstack to parent all
    5.724 DEBUG    Adding host s1-2-uk1 to group devstack_
    5.724 DEBUG    Adding child group devstack_ to parent all
    5.725 DEBUG    Adding host s1-2-uk1 to group devstack__nova
    5.725 DEBUG    Adding child group devstack__nova to parent all
    5.725 DEBUG    Adding host s1-2-uk1 to group instance-***
    5.725 DEBUG    Adding child group instance-**** to parent all
    5.726 DEBUG    Adding host s1-2-uk1 to group nova
    5.726 DEBUG    Adding child group nova to parent all
    5.726 INFO     Loaded 6 groups, 1 hosts
    5.732 DEBUG    Inventory variables unmodified
    5.744 DEBUG    Group "_nova" added
    5.749 DEBUG    Group "devstack" added
    5.752 DEBUG    Group "devstack_" added
    5.756 DEBUG    Group "devstack__nova" added
    5.759 DEBUG    Group "instance-***" added
    5.762 DEBUG    Group "nova" added
    5.772 DEBUG    Host "s1-2-uk1" added
    5.780 DEBUG    Host "s1-2-uk1" added to group "devstack"
    5.786 DEBUG    Host "s1-2-uk1" added to group "devstack_"
    5.791 DEBUG    Host "s1-2-uk1" added to group "devstack__nova"
    5.803 DEBUG    Host "s1-2-uk1" added to group "instance-****"
    5.849 DEBUG    Host "s1-2-uk1" added to group "_nova"
    5.857 DEBUG    Host "s1-2-uk1" added to group "nova"
    5.941 INFO     Inventory import completed for OVH integration in 0.2s

Some errors :

Not replacing invalid character(s) "{'-'}" in group name (instance-***)
[WARNING]: Invalid characters were found in group names but not replaced, use

Second test: Specify region UK1 as filter Curl all regions 1 server from uk1 0 server from GRA7 Get error 401 and warning

third test: Specify region GRA7 as filter Curl all regions 1 server from uk1 0 server from GRA7 Get error 401 and warning

I don't understand why I get 401 Unauthorized when I sync UK1 server ? Also why region name doesnt filter sync request to specified region ?

If you need more information, feel free to ask.

Thanks you

best regards

nifr commented 3 years ago

This issue might be caused by the openstack.cloud.openstack inventory plugin itself - not awx.

Can you reproduce the issue (locally) with the ansible-inventory command?

Procedure would be somewhat as follows (untested).

Create an openstack clouds.yml file - see config files

# clouds.yml
clouds:
  ovh:
    region_name: GRA7
    auth:
      username: 'admin'
      password: 'pw'
      project_name: 'my_project'
      auth_url: 'https://auth.cloud.ovh.net/v3'

Create an inventory config named openstack.yml and reference the clouds.yml under key clouds_yaml_path

# openstack.yml
plugin: 'openstack.cloud.openstack'
expand_hostvars: yes
fail_on_errors: yes
all_projects: no
clouds_yaml_path: /path/to/clouds.yml

Run ansible-inventory -i openstack.yml --list.

You can view the documentation and examples from the CLI with ansible-doc -t inventory openstack.cloud.openstack

bdoublet91 commented 3 years ago

HI,

Thanks for your answer.

I did your test. Created clouds.yml and openstack.yml

bash-4.4# ansible-inventory -vv -i openstack.yml --list
ansible-inventory 2.9.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-inventory
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file
[WARNING]:  * Failed to parse /tmp/openstack.yml with auto plugin: openstacksdk is required for the OpenStack inventory plugin. OpenStack inventory sources will be skipped.
[WARNING]:  * Failed to parse /tmp/openstack.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]:  * Failed to parse /tmp/openstack.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a
port.
[WARNING]: Unable to parse /tmp/openstack.yml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available

Install openstack.cloud.openstack with ansible galaxy and openstacksdk with pip3

ansible-galaxy collection install openstack.cloud
pip3 install openstacksdk

bash-4.4# ansible-inventory -vv -i openstack.yml --list
ansible-inventory 2.9.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-inventory
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file
[WARNING]:  * Failed to parse /tmp/openstack.yml with auto plugin: must be str, not list
[WARNING]:  * Failed to parse /tmp/openstack.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]:  * Failed to parse /tmp/openstack.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a
port.
[WARNING]: Unable to parse /tmp/openstack.yml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

clouds_yaml_path should be a list: https://github.com/ansible/ansible/issues/59442

# openstack.yml
plugin: openstack.cloud.openstack
expand_hostvars: yes
fail_on_errors: yes
all_projects: no
clouds_yaml_path:
  - /tmp/clouds.yml

Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400)

Specify domain in clouds.yml

# clouds.yml
clouds:
  ovh:
    region_name: GRA7
    auth:
      username: '***'
      password: '***'
      project_name: '***'
      domain_name: default
      auth_url: 'https://auth.cloud.ovh.net/v3'

And now, it successfull!

bash-4.4# ansible-inventory -vv -i openstack.yml --list
    "ovh_GRA7": {
        "hosts": [
            "int-HAproxy-001",
            "int-HAproxy-002",
            "int-administration-001",
            "int-console-001",
            "int-jenkins",
            "int-jenkins-002",
            "int-openvas-001",
            "int-postgres-001",
            "int-postgres-002",
            "int-postgres-003",
            "int-storage-001",
            "int-swarm-manager-001",
            "int-swarm-manager-002",
            "int-swarm-manager-003",
            "int-swarm-worker-001",
            "int-swarm-worker-002",
            "int-swarm-worker-003"
        ]
    },

So I tried to reload the job with awx but get the same result, no servers sync. I printed the output of /tmp/awx_***/openstack.yml and clouds.yml

clouds:
  devstack:
    auth:
      auth_url: https://auth.cloud.ovh.net/v3
      domain_name: default
      password: ***
      project_name: '***'
      username: ***
    private: true
    verify: false

and openstack.yml

bash-4.4# cat awx_***/openstack.yml 
plugin: openstack.cloud.openstack
bash-4.4#

no region name in config instead of I specified in awx credentials.

Remake a test with ansible-inventory without specified region_name and get no results

bash-4.4# ansible-inventory -vv -i openstack.yml --list
ansible-inventory 2.9.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-inventory
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /etc/ansible/ansible.cfg as config file
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

Problem with configuration file ? What do you think about it ?

If you need more information, feel free to ask.

Best regards

nifr commented 3 years ago

@bdoublet91 - do I understand this correctly?

There are basically 2 issues here:

awx doesn't write the region_name credential entered in the UI into the generated clouds.yml configuration
the openstack inventory plugin does not return any hosts if region_name is not set

btw you can limit the inventory plugins used with the ANSIBLE_INVENTORY_ENABLED environment variable. That way you get rid of the multiple parse error messages:

Failed to parse with plugin:

example:

ANSIBLE_INVENTORY_ENABLED=openstack.cloud.openstack \
ANSIBLE_INVENTORY_ANY_UNPARSED_IS_FAILED=1 \
  ansible-inventory \
    --inventory openstack.yml \
    --list

bdoublet91 commented 3 years ago

awx doesn't write the region_name credential entered in the UI into the generated clouds.yml configuration Yeah the configuration has no entry region_name with awx 17 May I test it with awx inventory cli and my configuration file already created with region name ? https://docs.ansible.com/ansible-tower/latest/html/towercli/index.html
the openstack inventory plugin does not return any hosts if region_name is not set I guess but I have never used this plugin before so further tests could be nice to validate this issue

I though that awx 16 requested all region_name when no one was set before version 17 ? But tested also on version 16 and awx only found server from uk1 region.

I'm the only one with this issue ? or someone managed it ?

ANSIBLE_INVENTORY_ENABLED=openstack.cloud.openstack \ ANSIBLE_INVENTORY_ANY_UNPARSED_IS_FAILED=1 \ ansible-inventory \ --inventory openstack.yml \ --list

it works without warning

wenottingham commented 3 years ago

awx doesn't write the region_name credential entered in the UI into the generated clouds.yml configuration

I think https://github.com/ansible/awx/pull/8880/files is broken.

The PR defines the field name as region_name on the model, but refers to it as project_region_name in the injector and tests.

CC @vguillard who contributed that change.

bdoublet91 commented 3 years ago

May I test it with awx inventory cli and my configuration file already created with region name ? https://docs.ansible.com/ansible-tower/latest/html/towercli/index.html ?

Did it work in version 16 ?

wenottingham commented 3 years ago

There was no support for specifying region at all in AWX before... so I doubt it ever worked.

bdoublet91 commented 3 years ago

Ok I will deploy awx 18.0.0, test and type a feedback

bdoublet91 commented 3 years ago

Hi, I tried to install awx 18 with docker installation but there are a lot of breaking change and no documentation because awx dev decided to recommand kubernetes over docker so I can't spend time to debug the devel installation. I keep awx 17.1.0 and wait for another one doing the test.

Hope a rollback of dev decision about docker ...

tiagodread commented 3 years ago

In AWX 19 I can't get openstack inv plugin working in a default execution env:

auto plugin: plugin config file, but not for us: openstack
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/auto.py", line 58, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/openstack/cloud/plugins/inventory/openstack.py", line 167, in parse
    raise AnsibleParserError(msg)
[WARNING]:  * Failed to parse /runner/project/inventories/openstack.yml with
yaml plugin: Plugin configuration YAML file, not YAML inventory
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/yaml.py", line 112, in parse
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]:  * Failed to parse /runner/project/inventories/openstack.yml with
ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not
allowed, this character is reserved to provide a port.
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/ini.py", line 136, in parse
    raise AnsibleParserError(e)
[WARNING]: Unable to parse /runner/project/inventories/openstack.yml as an
inventory source
ERROR! No inventory was parsed, please check your configuration and options.

cc: @AlanCoding @shanemcd

AlanCoding commented 3 years ago

Yes, its a known issue that the openstack inventory plugin doesn't work due to collection naming issues. Safe to say that blocks anyone working on this issue about the region not working.

wenottingham commented 3 years ago

Actually, closing this issue. The region fix landed, what you're seeing now is just #10043

HolmfryWrk commented 2 years ago

I'm still having the openstack plugin issues on awx 21 failed to parse /runner/inventory/openstack.yml with auto plugin region

EffectShapiro commented 2 years ago

Same as @HolmfryWrk using AWX 21.3.0

EffectShapiro commented 2 years ago

Here is my inventory variables

---
clouds:
  openstack:
    auth:
      auth_url: https://openstack-api.fqdn/identity/v3
      username: "myusername"
      project_id: ********************************
      project_name: "myproject"
      user_domain_name: "Default"
    region_name: "RegionOne"
    interface: "public"
    identity_api_version: 3

I specified the region name in the matching credentials as well

my inventory source

---
plugin: openstack
use_hostnames: true
expand_hostvars: true
fail_on_errors: true
private: true

Still having the problem

ansible-inventory [core 2.12.5.post0]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
  jinja version = 2.11.3
  libyaml = True
No config file found; using defaults
redirecting (type: inventory) ansible.builtin.openstack to openstack.cloud.openstack
[WARNING]:  * Failed to parse /runner/inventory/openstack.yml with auto plugin:
'region'
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 290, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/auto.py", line 58, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/openstack/cloud/plugins/inventory/openstack.py", line 263, in parse
    self._populate_from_source(source_data)
  File "/usr/share/ansible/collections/ansible_collections/openstack/cloud/plugins/inventory/openstack.py", line 281, in _populate_from_source
    self._append_hostvars(hostvars, groups, name, servers[0])
  File "/usr/share/ansible/collections/ansible_collections/openstack/cloud/plugins/inventory/openstack.py", line 392, in _append_hostvars
    for group in self._get_groups_from_server(server, namegroup=namegroup):

Thanks

HolmfryWrk commented 2 years ago

@EffectShapiro After I was clearly losing my mind, my colleague did get it working but it involved customizing the pod configuration for awx-ee, and using an alternate openstack plugin. I'm not sure of all the details, and he's on vacation this week but I've sent him an email asking him to respond when he returns if possible.

EffectShapiro commented 2 years ago

@HolmfryWrk Oh great ! Thank you very much for asking your colleague, i'll be really glad if he still has a trace of how to fix it 🙏 Sounds messy to customize pod configuration though but if it works.. Cheers

HolmfryWrk commented 2 years ago

You're very welcome! I tried to solve this for so long so I know how frustrating it is (esp. when all the bug reports about it say 'solved' for some reason..); I know it did seem messy but it can work lol

colinrobertmcculloch commented 2 years ago

@HolmfryWrk Oh great ! Thank you very much for asking your colleague, i'll be really glad if he still has a trace of how to fix it 🙏 Sounds messy to customize pod configuration though but if it works.. Cheers

(I'm Holmfry's colleague) I got it to work by modifying the awx-ee Containerfile to require a lower version of openstacksdk (it defaults to 0.99 but that's incompatible with older Openstack deployments) and then rebuilt the awx-ee image. The AWX files are available here. I added in some additional commands to install pip and included the openstacksdk files in ./awx-ee/_build/ and included the source in ./awx-ee/_build/requirements.yml

---
collections:
  - name: awx.awx
 ...
  - source: /build/ansible-collections-openstack
    type: dir

Lastly, in ./awx-ee/_build/ansible-collections-openstack/requirements.txt I changed openstacksdk>=0.99.0 to openstacksdk<0.99 This builds a custom awx-ee:0.6.0 image with an openstacksdk version that should be compatible with older Openstack deployments.

tomascribb10 commented 2 years ago

Hi colinrobertmcculloch, Very nice work!! Could you explain how did you replace the official awx-ee image for the custom one? I built the image with docker but I'm not able import to k3s cluster.

colinrobertmcculloch commented 2 years ago

We have a private container registry that I pushed the image to and changed my AWX resource definition to include the custom image. If your AWX deployment is already running you can add EE images to it but modifying the 'Kind: AWX' resource by adding the image as seen here.

tomascribb10 commented 2 years ago

Thank you very much @colinrobertmcculloch. I did some tests and finally it work. I share the steps that I did to use Docker external registry in case it helps to someone.

Install podman
git clone https://github.com/ansible/awx-ee.git
vim awx-ee/_build/requirements.yml `--- collections:
- name: awx.awx
- name: azure.azcollection
- name: amazon.aws
- name: theforeman.foreman
- name: google.cloud
- name: community.vmware
- name: ovirt.ovirt
- name: kubernetes.core
- name: ansible.posix
- name: ansible.windows
- name: redhatinsights.insights
- source: /build/ansible-collections-openstack type: dir `
cd awx-ee/_build/
git clone https://github.com/openstack/ansible-collections-openstack.git
vim awx-ee/_build/ansible-collections-openstack/requirements.txt openstacksdk<0.99.0
tox -epodman
podman push xxxxxxxxxxx docker://docker.io/xxxxx/awx-ee
In Web Interface - Create Execution Environment: Name: Custom-ee Image: docker.io/xxxxx/awx-ee
RUN Job.

This steps worked for me in AWX 21.4.0 and 21.0.0 with Openstack stein.

ansible / awx