Open HeroesLament opened 1 year ago
HeroesLament
commented
1 year ago Running locally on my Garuda Linux workstation works just fine, and on all my Ansible AWX EE containers running the latest version of Python 3.11.2 in the Redhat repos, we get this stack trace pointing to some regex search function. This is reproducible and I am available for testing any fixes.
HeroesLament
commented
1 year ago Same behavior using a debian-based Python 3.11.4 base image:
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /runner/project/tools/ansible-awx/netbox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/tools/ansible-awx/netbox.yml as it did not pass its verify_file() method
Loading collection netbox.netbox from /usr/share/ansible/collections/ansible_collections/netbox/netbox
Using inventory plugin 'ansible_collections.netbox.netbox.plugins.inventory.nb_inventory' to process inventory source '/runner/project/tools/ansible-awx/netbox.yml'
Fetching: http://netbox.alpha.gg:8008/api/status
toml declined parsing /runner/project/tools/ansible-awx/netbox.yml as it did not pass its verify_file() method
[WARNING]: * Failed to parse /runner/project/tools/ansible-awx/netbox.yml with
auto plugin: expected string or bytes-like object, got 'int'
File "/usr/local/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/usr/local/lib/python3.11/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
plugin.parse(inventory, loader, path, cache=cache)
File "/usr/share/ansible/collections/ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py", line 2039, in parse
self.main()
File "/usr/share/ansible/collections/ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py", line 1895, in main
self.fetch_api_docs()
File "/usr/share/ansible/collections/ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py", line 1501, in fetch_api_docs
if version.parse(netbox_api_version) >= version.parse("3.5.0"):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/version.py", line 52, in parse
return Version(version)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/packaging/version.py", line 196, in __init__
match = self._regex.search(version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
[WARNING]: * Failed to parse /runner/project/tools/ansible-awx/netbox.yml with
yaml plugin: Plugin configuration YAML file, not YAML inventory
File "/usr/local/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/usr/local/lib/python3.11/site-packages/ansible/plugins/inventory/yaml.py", line 114, in parse
raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]: * Failed to parse /runner/project/tools/ansible-awx/netbox.yml with
ini plugin: Invalid host pattern '---' supplied, '---' is normally a sign this
is a YAML file.
File "/usr/local/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/usr/local/lib/python3.11/site-packages/ansible/plugins/inventory/ini.py", line 137, in parse
raise AnsibleParserError(e)
[WARNING]: Unable to parse /runner/project/tools/ansible-awx/netbox.yml as an
inventory source
ERROR! No inventory was parsed, please check your configuration and options.
sc68cal
commented
1 year ago So, the code for fetch_api_docs catches all exceptions and sets the version to 0 which is not the correct thing to do. That value of 0 gets passed to version.parse and that's why it fails.
So, from our side we need to make that code better, and not just squash the error.
@HeroesLament there appears to be some sort of error occuring when the plugin attempts to access your netbox server's API doc endpoint at /api/status
HeroesLament
commented
1 year ago @sc68cal Thanks for the input, and I actually was able to fully drill down what was going on, including the error handling.
First, in my Ansible AWX EE container, I set a PDB breakpoint at line 1493, to step through what was going on. I stepped through till we got to the fetch /status/api statement, and then we get this:
Fetching: http://netbox.alpha.gg:8008/api/status
urllib.error.URLError: <urlopen error timed out>
> /usr/share/ansible/collections/ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py(1485)fetch_api_docs()
-> status = self._fetch_information(self.api_endpoint + "/api/status")
(Pdb) quitSo that clued me in that there was an HTTP reachability issue. I checked further, and it was actually an issue where the AWX controller and the Netbox containers were running on the same host, and they were unable to open a socket to the other container due to the way Docker publishes ports. I've worked around that by attaching the AWX Controller and the Netbox-Main containers to the same network, with a more elegant fix to come later. But for now, we are importing!
ansible-inventory [core 2.15.1]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
executable location = /usr/local/bin/ansible-inventory
python version = 3.11.4 (main, Jul 4 2023, 05:35:05) [GCC 12.2.0] (/usr/local/bin/python)
jinja version = 3.1.2
libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /runner/project/tools/ansible-awx/netbox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/tools/ansible-awx/netbox.yml as it did not pass its verify_file() method
Loading collection netbox.netbox from /usr/share/ansible/collections/ansible_collections/netbox/netbox
Using inventory plugin 'ansible_collections.netbox.netbox.plugins.inventory.nb_inventory' to process inventory source '/runner/project/tools/ansible-awx/netbox.yml'
Fetching: http://192.168.112.5:8080/api/status
Fetching: http://192.168.112.5:8080/api/docs/?format=openapi
Fetching: http://192.168.112.5:8080/api/dcim/devices/?limit=0&tag=production&exclude=config_context
Fetching: http://192.168.112.5:8080/api/virtualization/virtual-machines/?limit=0&tag=production&exclude=config_context
Fetching: http://192.168.112.5:8080/api/dcim/sites/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/regions/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/site-groups/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/locations/?limit=0
Fetching: http://192.168.112.5:8080/api/tenancy/tenants/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/device-roles/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/platforms/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/device-types/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/manufacturers/?limit=0
Fetching: http://192.168.112.5:8080/api/virtualization/clusters/?limit=0
Fetching: http://192.168.112.5:8080/api/ipam/services/?limit=0
Fetching: http://192.168.112.5:8080/api/dcim/racks/?limit=0
Parsed /runner/project/tools/ansible-awx/netbox.yml inventory source with auto plugin
52.822 INFO Processing JSON output...
52.826 DEBUG Loaded group: all
52.827 DEBUG Loaded group: tags_production
52.836 DEBUG Adding child group tags_production to parent all
52.843 DEBUG Loaded host: rtr-lhr-01
52.847 DEBUG Adding host rtr-lhr-01 to group tags_production
52.848 DEBUG Adding child group tags_production to parent all
52.866 INFO Loaded 1 groups, 1 hosts
52.871 DEBUG Dumping tree for group "all":
52.881 DEBUG - Vars: {}
52.882 DEBUG - Child: tags_production
52.883 DEBUG ----
52.883 DEBUG Dumping tree for group "tags_production":
52.883 DEBUG - Vars: {}
52.884 DEBUG - Host: rtr-lhr-01, {'cluster': 'London 1', 'cluster_type': 'vultr', 'custom_fields': {}, 'device_roles': ['router'], 'disk': 60, 'is_virtual': True, 'local_context_data': [None], 'locations': [], 'memory': 2048, 'regions': [], 'services': [], 'site_groups': [], 'sites': ['vultr-lhr-01'], 'status': {'label': 'Active', 'value': 'active'}, 'tags': ['production'], 'vcpus': 2.0}
52.884 DEBUG ----
52.885 WARNING loading into database...
52.899 DEBUG Inventory variables unmodified
52.994 DEBUG Group "tags_production" added
53.007 WARNING group updates took 9 queries for 1 groups
53.057 DEBUG Host "rtr-lhr-01" added
53.087 WARNING host updates took 8 queries for 1 hosts
53.090 WARNING Group-group updates took 0 queries for 0 group-group relationships
53.130 DEBUG Host "rtr-lhr-01" added to group "tags_production"
53.190 WARNING Group-host updates took 5 queries for 1 group-host relationships
53.352 WARNING update computed fields took 25 queries
53.361 WARNING Inventory import completed for Netbox Inventory Source in 0.5s
53.366 WARNING Inventory import required 52 queries taking 0.240sSo big win there!
If you don't mind, could I actually contribute a PR to add some error handling to the fetch_api_docs() function?
Thanks!
sc68cal
commented
1 year ago Yes please do add some better handling to that function. I will help review.
Ansible NetBox Collection version
v3.13.0
Ansible version
NetBox version
v3.4.7
Python version
3.11
Steps to Reproduce
ansible-inventory -i netbox.yml --list
Expected Behavior
Observed Behavior