Closed CGHoussem closed 8 months ago
Thanks to this answer in a stackoverflow post (https://stackoverflow.com/a/71266343), I found the solution.
A solution that worked for us without pinning old versions of dependencies was to add client_secret in https://github.com/ansible-collections/azure/blob/dev/plugins/inventory/azure_rm.py#L229:
def _credential_setup(self):
auth_options = dict(
...
client_secret=self.get_option('secret'),
...
)
This seems to be related to an upstream change in the auth library, where they now expect client_secret
instead of secret
(and, similarly, tenant_id
instead of tenant
---this may also need to be fixed).
Unfortunately, we run a fairly heavily modified version from what's in this repo, so I don't have a great way to submit and test a patch here, but I figured I'd drop this note in case it's of help!
I don't understand why this has been closed.
The error is still there in any version of azure.azcollection > 1.10.0
.
@xuzhang3 @Fred-sun Could the fix proposed by @ChandlerSwift be added to azure_rm.py
?
@fredflev how do you install the install the collection dependencies?
@xuzhang3
@fredflev how do you install the install the collection dependencies?
Hi, I use pip and install the 6.1.0
version of ansible via $(brew --prefix)/bin/python3 -mpip install -r requirements.txt --user
that pulls the 1.13.0
version of azure.azcollection
.
My requirements.txt
:
ansible==6.1.0
ansible-lint==5.3.2
azure-cli
azure-common
molecule>=3.5
molecule-docker
msrestazure
As mentioned, installing ansible>=6.1.0
also installs azure.azcollection 1.13.0
ansible-galaxy collection list | grep azure
azure.azcollection 1.13.0
community.azure 1.1.0
netapp.azure 21.10.0
I think one of the issue here is that I install the latest version of azure-cli
(2.39.0) as part of the requirements. ( I need it for az login
etc..)
And the requirements of azure-cli 2.39.0
and azcollection 1.13.0
seem to mismatch by miles.
https://github.com/ansible-collections/azure/blob/v1.13.0/requirements-azure.txt
So the "old" packages used by azcollection 1.13.0
are probably installed first when I am installing ansible==6.1.0
and then updated when I install the latest version of azure-cli
.
So if I then try to install the actual azcollection 1.13.0
requirements via :
pip3 install -r /usr/local/lib/python3.9/site-packages/ansible_collections/azure/azcollection/requirement.txt
It will downgrade most of the dependencies I need for azure cli 2.39.0
.
And obviously running the command above makes azure-cli
installation unusable since it can't find the required dependencies anymore.
The quickest way to recover this is to uninstall all my pip packages via pip3 uninstall -y -r <(pip3 freeze)
and revert to using ansible-5.2.0
https://github.com/ansible-community/ansible-build-data/blob/main/5/ansible-5.2.0.deps in
my requirements.txt
file
Things work has expected with ansible==5.2.0
since it relies on azcollection 1.10.0
that doesn't have the 'client_secret' is not defined
bug when using azure_rm.py
.
So this works fine:
ansible==5.2.0
ansible-lint==5.3.2
azure-cli
azure-common
datadog
molecule>=3.5
molecule-docker
msrestazure
So in short the issues here is that the list of dependencies in https://github.com/ansible-collections/azure/blob/v1.13.0/requirements-azure.txt is really outdated and that azcollection 1.13.0
can't work with any version of azure-mgmt-authorization > 0.51.1
(the all client_secret
issue stuff).
The "requirements mismatch" between azure-cli
and azcollection
seems to be a well known issue:
For anyone who's facing the same issue:
I ended up installing azure-cli
in the default OS python env via python3 -m pip install azure-cli --user
and running ansible + azcollection and its dependencies in a dedicated virtual env using pipenv
Below is an example of my Pipfile
for ansible 6.4.0
( It comes with Azcollection version 1.13.0
)
# Dependencies specific to azcollection in this file can be updated by running the scripts defined in [scripts]
# `pipenv run azcollection_req_update`
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[packages]
ansible = "==6.4"
ansible-lint = "~=6.7"
datadog = "*"
molecule-docker = "~=2.1"
# The packages below are required by azcollection
azure-cli-core = "==2.34.0"
azure-common = "==1.1.11"
azure-graphrbac = "==0.61.1"
azure-identity = "==1.7.0"
azure-keyvault = "==1.0.0a1"
azure-mgmt-apimanagement = "==0.2.0"
azure-mgmt-authorization = "==0.51.1"
azure-mgmt-automation = "==1.0.0"
azure-mgmt-batch = "==5.0.1"
azure-mgmt-cdn = "==3.0.0"
azure-mgmt-compute = "==26.1.0"
azure-mgmt-containerinstance = "==1.4.0"
azure-mgmt-containerregistry = "==2.0.0"
azure-mgmt-containerservice = "==9.1.0"
azure-mgmt-cosmosdb = "==0.15.0"
azure-mgmt-datafactory = "==2.0.0"
azure-mgmt-datalake-store = "==0.5.0"
azure-mgmt-devtestlabs = "==3.0.0"
azure-mgmt-dns = "==2.1.0"
azure-mgmt-eventhub = "==2.0.0"
azure-mgmt-hdinsight = "==0.1.0"
azure-mgmt-iothub = "==0.7.0"
azure-mgmt-keyvault = "==1.1.0"
azure-mgmt-loganalytics = "==1.0.0"
azure-mgmt-managedservices = "==1.0.0"
azure-mgmt-managementgroups = "==0.2.0"
azure-mgmt-marketplaceordering = "==0.1.0"
azure-mgmt-monitor = "==3.0.0"
azure-mgmt-network = "==19.1.0"
azure-mgmt-notificationhubs = "==2.0.0"
azure-mgmt-nspkg = "==2.0.0"
azure-mgmt-privatedns = "==0.1.0"
azure-mgmt-rdbms = "==1.9.0"
azure-mgmt-recoveryservices = "==0.4.0"
azure-mgmt-recoveryservicesbackup = "==0.6.0"
azure-mgmt-redis = "==13.0.0"
azure-mgmt-resource = "==10.2.0"
azure-mgmt-search = "==3.0.0"
azure-mgmt-servicebus = "==0.5.3"
azure-mgmt-sql = "==3.0.1"
azure-mgmt-storage = "==19.0.0"
azure-mgmt-trafficmanager = "==0.50.0"
azure-mgmt-web = "==0.41.0"
azure-nspkg = "==2.0.0"
azure-storage = "==0.35.1"
msrest = "==0.6.21"
msrestazure = "==0.6.4"
packaging = "*"
xmltodict = "*"
requests = {extras = ["security"], version = "*"}
[dev-packages]
[requires]
python_version = "3.10"
[scripts]
# The version of python in the path below should matched the one defined in the [requires] section above.
azcollection_req_update = "pipenv install -r .venv/lib/python3.10/site-packages/ansible_collections/azure/azcollection/requirements-azure.txt"
The only draw back of this approach is that now I need to activate the virtualenv prior to run any ansible command but that's fine by me.
pipenv shell
ansible -i my_inventory -m ping all
@CGHoussem does this issue still exist?
Following the thread, apparently it does still exist. I've closed the issue before since there are alternative solutions. That being said, the issue still persists in the default installation.
@CGHoussem tried with latest release v1.13.0 , this issue has been fixed.
@xuzhang3 Unfortunately I can no longer test the issue (I have no azure subscription); Therefore, I will take your word for it. Thank you for the confirmation! 👍
I've installed azure.azcollection 1.14.0, but the error is still there:
[WARNING]: * Failed to parse ... with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: name 'client_secret' is not defined
% ansible --version
ansible [core 2.13.7]
% ansible-galaxy collection list
# /Users/kenorb/.ansible/collections/ansible_collections
Collection Version
------------------ -------
azure.azcollection 1.14.0
Even when using within pipenv shell
.
Details:
% ansible-inventory -i foo --graph -vvv
ansible-inventory [core 2.13.5]
config file = /Users/kenorb/ansible/ansible.cfg
configured module search path = ['/Users/kenorb/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/homebrew/lib/python3.10/site-packages/ansible
ansible collection location = /Users/kenorb/.ansible/collections:/usr/share/ansible/collections
executable location = /opt/homebrew/bin/ansible-inventory
python version = 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]
jinja version = 3.1.2
libyaml = True
...
redirecting (type: inventory) ansible.builtin.azure_rm to azure.azcollection.azure_rm
script declined parsing /Users/kenorb/ansible/foo/01-azure_rm.yml as it did not pass its verify_file() method
[WARNING]: * Failed to parse /Users/kenorb/ansible/foo/01-azure_rm.yml with ansible_collections.azure.azcollection.plugins.inventory.azure_rm
plugin: name 'client_secret' is not defined
Stack trace:
[WARNING]: * Failed to parse ansible/foo/01-azure_rm.yml with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: name
'client_secret' is not defined
File "/opt/homebrew/lib/python3.10/site-packages/ansible/inventory/manager.py", line 290, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/Users/kenorb/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 220, in parse
self._credential_setup()
File "/Users/kenorb/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 241, in _credential_setup
self.azure_auth = AzureRMAuth(**auth_options)
File "/Users/kenorb/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1546, in __init__
self.azure_credential_track2 = client_secret.ClientSecretCredential(client_id=self.credentials['client_id'],
I think it's the problem of azure-identity
(called from client_secret.ClientSecretCredential
in module_utils/azure_rm_common.py
). Including the latest version (pip3 install azure-identity==1.12.0
) and 1.11.0. It's probably loading client_secret
instead of secret
from the Azure's credentials file.
Temporary workaround was to downgrade Azure collection to 1.10.0
. Issue is happening between 1.11.0
and 1.14.0
(included).
ansible-galaxy collection install azure.azcollection:=1.10.0
See comparison for v1.10.0...v1.11.0 (search for client_secret
).
For the better solution, check further analysis below.
Ok, I've found the issue. All of below changes are in azure/azcollection/plugins/module_utils/azure_rm_common.py
.
The following import of client_secret
(which fails) is wrapper in try
/except
block:
try:
...
from azure.identity._credentials import client_secret, user_password, certificate
except ImportError as exc:
...
When tested import line (as it is) manually in python3
shell, it's loaded fine. So the only way it's not loaded properly, is due to previous import error. Since we don't print any errors in except
block, I had to manually print it via print(exc)
. In general I'm not sure why a lot of modules fail silently to import.
except ImportError as exc:
print(exc)
then re-execute ansible-inventory -i foo --graph -vvv
to see the failed module.
So the real error was with a different module:
No module named 'azure.storage.blob'
I've got the confirmation from Python3 shell:
% python3
>>> from azure.storage.blob import BlobServiceClient
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'azure.storage.blob'
>>> import azure.storage
(this loads fine)
Fixed by: pip3 install azure-storage-blob
The same error happens with azure.mgmt.automation
:
No module named 'azure.mgmt.automation'
Fixed by: pip3 install azure-mgmt-automation
And another one:
cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' (/opt/homebrew/lib/python3.10/site-packages/azure/mgmt/recoveryservicesbackup/init.py)
Attempt fix:
% pip3 install azure-mgmt-recoveryservicesbackup
Requirement already satisfied: azure-mgmt-recoveryservicesbackup in /opt/homebrew/lib/python3.10/site-packages (5.0.0)
Didn't help.
and next one:
No module named 'azure.mgmt.recoveryservicesbackup.models'
Attempt fix:
% pip3 install azure-mgmt-recoveryservicesbackup
Requirement already satisfied: azure-mgmt-recoveryservicesbackup in /opt/homebrew/lib/python3.10/site-packages (5.0.0)
Didn't help, maybe the wrong version is installed.
Another:
No module named 'azure.mgmt.notificationhubs'
Fixed by:
% pip3 install azure-mgmt-notificationhubs
Successfully installed azure-mgmt-notificationhubs-8.0.0
At this point, I've tried to install all azure-mgmt
, but didn't work as expected:
% pip3 install azure-mgmt
Collecting azure-mgmt
Downloading azure-mgmt-5.0.0.zip (4.8 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
...
Starting with v5.0.0, the 'azure-mgmt' meta-package is deprecated and cannot be installed anymore.
Please install the service specific packages prefixed by `azure` needed for your application.
Another one missing:
% pip3 install azure-mgmt-datafactory
Successfully installed azure-mgmt-datafactory-2.10.0
The universal workaround in my case was to move import of client_secret
to the top of the list (after try
). Alternatively moving it outside of try
block. Alternatively (in case of more issues), install all the missing modules as I did above (add print(exc)
line to print which are actually failing).
For quicker installing missing/broken dependencies, use the following command (change tag version if necessary):
pip3 install -r https://raw.githubusercontent.com/ansible-collections/azure/v1.14.0/requirements-azure.txt
This should potentially fix all the missing modules (unless there are some conflicts which needs to be resolved manually). Not sure why this was not installed during installation of azure.azcollection
(or it could be not part of the process). Maybe during downgrading and upgrading I hit some kind of dependency conflicts.
Ok, it broke again after I've installed azure-cli
(pip3 install azure-cli
). It seems when installing azure-cli (latest stable 2.40.0), it's reinstalling all of the dependencies previously installed by ansible-collections' requirements-azure.txt
and removing some of them.
Then when I've installed requirements-azure.txt
right after azure-cli
, I've got the following dependency warnings:
% pip3 install -r https://raw.githubusercontent.com/ansible-collections/azure/v1.14.0/requirements-azure.txt
...
Installing collected packages: ...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
azure-mgmt-policyinsights 1.1.0b3 requires azure-mgmt-core<2.0.0,>=1.3.2, but you have azure-mgmt-core 1.3.0 which is incompatible.
azure-mgmt-netapp 8.1.0 requires azure-mgmt-core<2.0.0,>=1.3.1, but you have azure-mgmt-core 1.3.0 which is incompatible.
azure-mgmt-msi 6.1.0 requires azure-mgmt-core<2.0.0,>=1.3.1, but you have azure-mgmt-core 1.3.0 which is incompatible.
azure-mgmt-datalake-nspkg 3.0.1 requires azure-mgmt-nspkg>=3.0.0, but you have azure-mgmt-nspkg 2.0.0 which is incompatible.
azure-cli 2.40.0 requires azure-cli-core==2.40.0, but you have azure-cli-core 2.34.0 which is incompatible.
azure-cli 2.40.0 requires azure-graphrbac~=0.60.0, but you have azure-graphrbac 0.61.1 which is incompatible.
azure-cli 2.40.0 requires azure-mgmt-apimanagement~=3.0.0, but you have azure-mgmt-apimanagement 0.2.0 which is incompatible.
...
So if you need azure-cli
, you've to install it first (use it, e.g. to login), then install requirements-azure.txt
on top of it.
% pip3 install azure-cli
% pip3 install -r https://raw.githubusercontent.com/ansible-collections/azure/v1.14.0/requirements-azure.txt
It seems the current dependency requirements from v1.14.0 breaks dependencies of azure-cli:
% az login
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/homebrew/lib/python3.10/site-packages/azure/cli/__main__.py", line 16, in <module>
from azure.cli.core.intercept_survey import prompt_survey_message
ModuleNotFoundError: No module named 'azure.cli.core.intercept_survey'
So in summary if I install azure-cli
(2.40.0), it breaks previously installed dependencies of ansible-collections (v1.14.0) by showing client_secret
error first, but in reality it's failing on other missing modules (which were removed by installing azure-cli
).
@kenorb can you try install/run the azcollection a virtual environment?
@kenorb The azure-requirements.txt dependencies conflict with the azure-cli. There is no effective way to avoid such problems for the time being! If you can, only have it run in two separate virtual environments. Both will work well. As for the above mentioned dynamic host acquisition failure has been solved in the new version, you can upgrade to the latest version to try! Thanks!
any work on this? this is happening to me today with azure.azcollection 1.16 and ansible 8.1.0 and azure-cli 2.49.0 , with the auth type of 'auto', if i set it to cli it works i guess
Any update with this issue for azure.azcollection 1.17, getting the same issue with client secret not being found
@JLP2000 Can you share your error information? This will help solve your problem, thank you!
@Fred-sun This is the error, a envs file is being passed where "AZURE_SECRET" is assigned but still getting the error:
toml declined parsing /ansible/azstack.azure_rm.yml as it did not pass its verify_file() method
[WARNING]: * Failed to parse /ansible/azstack.azure_rm.yml with script plugin: problem running
/ansible/azstack.azure_rm.yml --list ([Errno 8] Exec format error: '/ansible/azstack.azure_rm.yml')
File "/usr/local/lib/python3.9/site-packages/ansible/inventory/manager.py", line 293, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/usr/local/lib/python3.9/site-packages/ansible/plugins/inventory/script.py", line 150, in parse
raise AnsibleParserError(to_native(e))
[WARNING]: * Failed to parse /ansible/azstack.azure_rm.yml with auto plugin: name 'client_secret' is not
defined
File "/usr/local/lib/python3.9/site-packages/ansible/inventory/manager.py", line 293, in parse_source
plugin.parse(self._inventory, self._loader, source, cache=cache)
File "/usr/local/lib/python3.9/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
plugin.parse(inventory, loader, path, cache=cache)
File "/root/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 223, in parse
self._credential_setup()
File "/root/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 244, in _credential_setup
self.azure_auth = AzureRMAuth(**auth_options)
File "/root/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1581, in __init__
self.azure_credential_track2 = client_secret.ClientSecretCredential(client_id=self.credentials['client_id'],
I am facing the same issue with ubuntu20.04 ,
ansible-inventory [core 2.12.3] config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /root/.local/lib/python3.8/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible-inventory python version = 3.8.10 (default, Nov 22 2023, 10:22:35) [GCC 9.4.0] jinja version = 3.1.2 libyaml = True
root@bad2e417f11a:/etc/ansible/inventory# ansible-galaxy collection list
Collection Version
azure.azcollection 2.1.1 root@bad2e417f11a:/etc/ansible/inventory# ansible --version ansible [core 2.12.3] config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /root/.local/lib/python3.8/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.8.10 (default, Nov 22 2023, 10:22:35) [GCC 9.4.0] jinja version = 3.1.2 libyaml = True
@kenorb @CGHoussem @sloharadmin azure.azcollection upgrades its credentials from trunck1 SDK to trunck2 SDK through the azure-identity file, so if you are upgrading to the new version, you need to install azure-identity. Link (https://github.com/ansible-collections/azure)specifies that all dependencies need to be installed! Thank you!
Requirements
ansible version >= 2.14
To install Azure collection hosted in Galaxy:
ansible-galaxy collection install azure.azcollection
Install dependencies required by the collection (adjust path to collection if necessary):
pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
To upgrade to the latest version of Azure collection:
ansible-galaxy collection install azure.azcollection --force
Any feedback?
I will close it if you have no other question! Wellcom to contribute new when you meet new errors! Thank you very much!
Experiencing the same error in version azure.azcollection 2.7.0... Anyone?
@lucasscheepers can you share the errors logs?
SUMMARY
ISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
COLLECTION VERSION
CONFIGURATION
OS / ENVIRONMENT
STEPS TO REPRODUCE
My azure_rm.yml file:
my ~/.azure/credentials file:
The command that I ran:
ansible-inventory -i azure_rm.yml --list
EXPECTED RESULTS
I should have the list of my inventory since I am trying to create a dynamic one. Keep in mind that when I set the "auth_source" to "cli", it works just fine but not when I use the other sources. For the moment I am running this from a WSL (ubuntu) then I am going to be running this from an azure pipeline.
ACTUAL RESULTS
The output explains itself 😞 I get this warning at first:
[WARNING]: * Failed to parse /home/houssem/TestProject/azure_rm.yml with auto plugin: name 'client_secret' is not defined