Open EdificomSA opened 4 months ago
I'm not sure this error is the same as the issues you linked. The standard modules and the lookup module use pynautobot, which has the retries
argument available. The inventory modules, however, use Ansible's standard open_url
function.
Secondly, HTTP or Connection errors should raise during the section of the code above where your error got raised: https://github.com/nautobot/nautobot-ansible/blob/900395a17d199fe1816a887fee4cc7992a01e56a/plugins/inventory/inventory.py#L296-L305
This leads me to believe it got a response, but the response was incomplete in some way.
So, not as an easy fix as just enabling retries. Not impossible, but will probably take extra work to implement.
You're right, it's probably not linked to the mentioned issue
I checked the Retry from urllib3 and in fact it does not behave with IncompleteRead exeception https://github.com/nautobot/pynautobot/blob/develop/pynautobot/core/api.py#L91
So somehow sometimes fetching the api spec at https://nautobot.xxx/api/docs/?format=openapi will return an IncompleteRead
I'll try solve the issue at the core meaning Nautobot API response, unsure where to look at tho (timeout ?)
I'm not 100% sure yet, but it appears that the Inventory plugin timeout default theoretical 60s value is not set anywhere in https://github.com/nautobot/nautobot-ansible/blob/develop/plugins/inventory/inventory.py and relies on module_utils
open_url default timeout of 10sec
https://github.com/ansible/ansible/blob/stable-2.15/lib/ansible/module_utils/urls.py#L1673
it appears that the Inventory plugin timeout default theoretical 60s value is not set anywhere
The default 60s is indeed set on the documentation on line 169. Inventory plugins are "documentable" plugins meaning the options are set automatically via the yaml styled documentation rather than needing to set them in python like you do for standard ansible modules.
I was able to confirm this by adding the following line to my local dev environment:
self.timeout = self.get_option("timeout")
self.display.v("timeout: %s" % self.timeout)
And then I was able to see it in the output:
# inventory.yml
---
plugin: networktocode.nautobot.inventory
api_endpoint: https://demo.nautobot.com/
token: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
❯ ansible-inventory -i inventory.yml --list -vvv
...
Using /root/nautobot-ansible/ansible.cfg as config file
timeout: 60
Fetching: https://demo.nautobot.com/api/docs/?format=openapi
...
ISSUE TYPE
SOFTWARE VERSIONS
pynautobot
2.2.1
Ansible:
2.15.9
Nautobot:
2.2.6
Collection:
networktocode.nautobot 5.2.1
SUMMARY
AWX inventory sync randomly fails
STEPS TO REPRODUCE
EXPECTED RESULTS
Based on idea from #367, might be a good idea to add a retry on error in
fetch_api_docs
like in #338 for lookupACTUAL RESULTS