Open ljosten opened 4 years ago
Fred-sun
commented
4 years ago @ljosten Do you still have this problem? I was checking the azure_rm.py module recently and I also tested it. I haven't encountered the problem you mentioned. Can you try again locally? And update the results of your test. Thank you!
ljosten
commented
4 years ago @Fred-sun the issue is occuring as of right now. It occurs in azure pipelines aswell as from my local setup. Here a sample output from a few minutes ago:
azure_rm.yml:
plugin: azure_rm
include_vm_resource_groups:
- s-example_40405977-sbx-rg
auth_source: auto
keyed_groups:
- prefix: tag
key: tagsansible-inventory -i azure_rm.yml:
{
"_meta": {
"hostvars": {}
},
"all": {
"children": [
"ungrouped"
]
}
}ansible-playbook -i azure_rm.yml site.yml:
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [Configure *****] ****************************************
skipping: no hosts matched
PLAY RECAP *********************************************************************
Finishing: Run Ansible rollout on *******The exact same configurations worked without problems only minutes before and returned a valid inventory. As I stated above the behaviour and rate of occurence changes during the time of the day, possibly due to heavier load on Azure API endpoints. A retry mechanism, API error return codes or a validation mechanism needs to be present to check the inventory before processing the playbook.
ljosten
commented
3 years ago Is there any progress here or further information required? The issue persists as of today, using ansible 2.10
kgorskowski
commented
3 years ago I can confirm this is happening periodically in our pipelines. After some retries the inventory is eventually generated but no consistent behavior
kgorskowski
commented
3 years ago We had some feedback from Azure side that this may be related to replication times between regions. We run ansible right after the rollout of the instances with terraform. So its possible according to the support that the state of the instance or resource group is not yet available at the first few ansible runs. We use "westeurope" as our default location for terraform provider configuration and the azure inventory plugin with the default settings, which resolves from my understanding to cloud_environment = AzureCloud. Is there a way to configure the azure plugin "talk" to the same region in which the instances were created in maybe?
matt961
commented
3 years ago Can report that I recently discovered this issue in a production deploy where one of my machines did not get picked up by the plugin, and the playbook skipped all tasks as a result. As there are technically no failures, my rolling release script happily puts back un-updated instances into the load balancer, resulting in two different versions of my code running. Without any changes I run my pipeline again and the same instance is added to the inventory as normal.
For me it is not (I would hope) a replication delay, as these machines have been deployed with the same tags for probably close to half a year now. That still doesn't explain why the very next run ~10 mins later it picks up both machines successfully.
To avoid potentially missing updates on machines I will be adding check tasks in my playbook that target localhost to make sure the relevant host groups have at least one host in them before all other tasks, and fail if not.
I'm on one of the later patches of ansible 2.9 using python 2.7.
ghost
commented
3 years ago I'm also seeing this issue. The majority of the time I'm seeing all the hosts being picked up correctly, but occasionally it'll only pick up a subset of them. When this happens, once I rerun the playbook, it will usually pick up all the hosts again correctly (with no changes at all.)
Originally I thought it was after terraform had updated our VM instances that we were seeing this behaviour, but I've noticed it also seems to happen even when there's been no recent updates to the VMs.
I'm using ansible 2.10.7 on Python 3.6.8
kgorskowski
commented
3 years ago Hey folks, just for completion. In our case it was a mix of network topology and a short delay in the sync of the resources between regions. We created the resources in one region but due to some background networking foo on our selfhosted agents side the request from the azure dynamic inventory went to a different region. In the end we were able to mitigate the network issue on our side and haven't seen this problem again so far.
ljosten
commented
3 years ago Also, in addition to Karsten's information, another problem with returning empty inventories was solved by passing an empty host filter list instead of using the default:
plugin: azure_rm
include_vm_resource_groups:
- sb8f2294b-xx1-sbx-rg
- sb8f2294b-xx2-sbx-rg
- sb8f2294b-xx3-sbx-rg
- sb8f2294b-xx4-sbx-rg
auth_source: auto
keyed_groups:
- prefix: tag
key: tags
default_host_filters: []We think this had to do with policies enforced in azure that idled for prolonged time and set the deployment status of nodes into a filtered state.
SUMMARY
When requesting an inventory using the dynamic azure_rm plugin no hosts are returned. All included resource groups include virtual machines up and running. This changes during the day with the same inventory yml configuration in use, from no hosts returned to some hosts returned to all hosts returned.
ISSUE TYPE
COMPONENT NAME
azure_rm dynamic inventory plugin
ANSIBLE VERSION
CONFIGURATION
OS / ENVIRONMENT
Ubuntu 20.04, RHEL 7 Installed via pip using the ansible[azure] collection
STEPS TO REPRODUCE
Create inventory definition, for example ansible_azure_rm.yml with following content
use ansible-inventory to list hosts:
EXPECTED RESULTS
hosts are retrieved and inventory is generated
ACTUAL RESULTS