azcollection requirements are so old, which azure-cli package will work with them?

[howardjones]

I'm trying to build a virtualenv for awx/tower that contains azcollection and also the azure cli (since playbooks still need that to do things that the azcollection can't, like creating Service Principals)

However, the current azcli uses a very short script wrapper over python modules python3 -Im azure.cli $@

The versions (2.11.1 vs 2.21.0 currently) installed by azcollection are not structured this way:

/var/lib/awx/venv/local/bin/python3: No module named azure.cli.__main__; 'azure.cli' is a package and cannot be directly executed

Which version of azure-cli will actually work with these old azure modules, until you move everything to the newer modules?

[howardjones]

Even cooler - there doesn't appear to be ANY version of azure-cli that matches the requirements for azcollection! azcollection wants azure-mgmt-batch==5.0.1

azure-cli-2.0.68 wants 6.0 azure-cli-2.0.67 wants 4.0.3

(both of which are nearly two years old at this point)

[Fred-sun]

@howardjones Thanks for your submit this issue! We are preparing to update all depend packages! once upgrade, I will add comment in here! Thank you very much!

[greystitch]

@howardjones Thanks for your submit this issue! We are preparing to update all depend packages! once upgrade, I will add comment in here! Thank you very much!

Does this mean that azcollection will be having the same python module dependencies as azure-cli defined somewhere in recommended requirements file?

[howardjones]

I was wondering this too. At least the same as some version of azure-cli. Currently with the new pip version resolver, pip will just churn and churn forever if you ask it for the azcollection dependencies and azure-cli, since they don't align.

[greystitch]

@howardjones Absolutely! And, as for now, the only solution to combine azcollection and azure-cli on the same machine (or container in my case) is to install CLI binary as package. But this workaround cannot be considered as good, because installation size is about 800MB and contains many other unwanted stuff, like another python installation, its libraries and hundreds of pyc files... Well, I'm really looking forward to see Ansible related functionality aligned with azure-cli, which might lift up the configuration management to the next level.

[paultaiton]

I'm not involved in the update that @Fred-sun mentioned, which is the way forward. However I wanted to know if using two virtual environments, one for azcollection, one for azure-cli, would be possible and mitigate the issue for you guys @howardjones and @greystitch in the meantime? There's no hard technical requirement that they both be installed together since azcollection does not directly call the cli. It is what I have done on my machines, although I do azcollection requirements to a venv and azure-cli to the default --user paths. You can then set "ansible_python_interpreter" to the venv during plays that require azcollection, or if using AWX / Tower you can follow those specific instructions for making virtual environments without a need for azure-cli.

[howardjones]

This is what I'm doing currently, but it's not very efficient... the second venv is another 650MB inside the container once azure-cli is installed with pip.

[paultaiton]

This is what I'm doing currently, but it's not very efficient... the second venv is another 650MB inside the container once azure-cli is installed with pip.

Gotcha. Yeah, not an ideal situation to duplicate so many packages for version differences.

[erikhjensen]

@paultaiton Hi, I'm being asked to look into this for another team where they do use the Azure Modules in some Tasks and they use the Azure CLI in others. When you commented that you're splitting up the virtual environments are you calling into other job-templates each with suitable Venvs from a master job-template? From what I gather, the team I'm working with is executing their job directly on the Ansible Worker and is looking for a venv with both capabilities.

[paultaiton]

@erikhjensen I do not call Azure CLI from Ansible at any time, and only use the azcollection modules, so my experience there is limited. I have azure cli installed to my user's default python path, and a dedicated .venv for azcollection. Any az command I run is done manually from a regular bash session.

In theory I believe you should be able to use the fully qualified path to the az executable, and it will pick the right python interpreter environment. That's probably the easiest way to get it working with both.

The other option I can think of is to change the "ansible_python_interpreter" on a per play basis, and keep your azcollection tasks and "az" tasks in separate plays.

[howardjones]

@erikhjensen I am doing this with awx at the moment - two extra venvs, one for azcollection (and whatever else we need), and a second for azcli. I also change the az "binary" shellscript to explicitly use the right venv's python instead of the discovery thing it does otherwise:

#!/usr/bin/env bash
AZ_INSTALLER=DEB /var/lib/awx/venv/azlocal/bin/python3 -Im azure.cli "$@"

so the job template runs in the azcollection venv, but it calls binaries in the other one which will use their local modules, since they are using that python.

If it's for local use, it's worth noting that the package-installed (i.e. apt not pip-installed) azure-cli on Ubuntu from the MS PPA does all this for you in /opt/az. Presumably it does the same in other packages they distribute.

[erikhjensen]

Hi @Fred-sun, how goes the planning of the dependency updates? I only ask because we had to back out a change which attempted to work around this limitation and I suspect our team is not quite as resourceful as @howardjones as our yum-installed Azure CLI was failing when called from within the awx virtual environments. Fwiw, this is the error we saw within our ansible_az virtual environment.

"stderr": "/var/lib/awx/venv/ansible_az/bin/python3: No module named azure.cli.__main__; 'azure.cli' is a package and cannot be directly executed",

[Fred-sun]

@erikhjensen In general, we install the azcollection dependencies through "pip install-r requires-azure. txt", you can refer to the link --"https://github.com/ansible-collections/azure", Thank you very much!

[erikhjensen]

Hi @Fred-sun, noted. I believe the concern in the thread is that this list of dependencies is falling far behind the dependencies in Azure CLI's python package and the two packages cannot be installed into the same virtual environment. We will monitor this thread for news of updates to the dependencies.

[egilh]

Installing requirements-azure.txt breaks azure-cli installation on MacOS, and after finding this issue, I guess we can assume that, yes - azcollection requirements are now so old, that installing Ansible with azcollection now breaks azure-cli on MacOS. I have been looking in to this issue as this has been working for me up to this week and upgrading homebrew packages. See comment (https://github.com/Azure/azure-cli/issues/19027#issuecomment-900866624 ) and (https://github.com/Azure/azure-cli/issues/19027#issuecomment-901127132

For me this renders this collection absolutely useless for local development as I require to use azure-cli for several other things and cannot rely on pip requirements alone. Running different virtual environments just to be able to use ansible against azure is also not an option for us.

This issue has already been open since April 1st and nothing seems to have been done to just bump these requirements. Requirements needs to follow these releases closer. Some of them are updated weekly.

[Liquidmantis]

I've been struggling with this for several days, too. I assumed it was something with my environment. Effectively this means I can't use azcollection from Azure Devops as my agent can't login when the azcollection requirements are installed.

[howardjones]

@Liquidmantis You can, but you would need to jump through some hoops with virtualenvs to do it (to run one or both of these tools in its own python venv). This works for us in awx, for similar uses. But it is no fun at all!

[Liquidmantis]

@howardjones Thanks! Yeah, I thought a solution might exist down that path, but that's more than I can take on currently :) Trying to get venvs working well on top of the Azure dynamic inventory plugin doesn't sound like a good time. Fortunately for us, our use case is almost entirely using Ansible on servers, not actual Azure management (so far). Terraform serves that role for us.

[geekq]

Based on my experience with azcollection I've solved the problem with obsolete dependencies and Azure API moving target by focusing on the single universal resource oriented API (compare it to az resource). You can read the motivation and try out my solution here: https://github.com/geekq/azbare Since it only has got 4 (instead of about 20) azure dependencies, it can be easily used after pip install azure-cli.

I was not sure, if issues tracker of azcollection is an appropriate place to promote an alternative solution ;-) @Fred-sun Maybe this resource API based approach can be one day adopted by the azcollection. The only alternative for azcollection to become maintainable and survive midterm would be to completely rework it and 100% generate the modules automatically (similar to the way e.g. terraform azure modules or azure SDK for different languages are done), but this is likely a much bigger effort

[l3ender]

@Fred-sun - any progress to share for updating the dependencies? If you are not able to work on it yourself, perhaps you could share advice/direction here and others will be able to assist. Thank you!

[l3ender]

Hi @Fred-sun, any update? Perhaps you could share advice on what needs to be done and others (myself included) will be able to assist.

[geekq]

Implementation note: many years ago the whole Azure SDK was contained in the single Microsoft library, azcollection was originally implemented by Microsoft employees, the central azcollection/plugins/module_utils/azure_rm_common.py depended on that single Azure SDK (azure pip - up to version 5.0.0).

Some time later Microsoft split they original SDK into multiple pips, see https://pypi.org/project/azure/ and all the different PIPs (divided by "provider", e.g. compute vs. rdbms.postgresql) are now developed and released at different pace. So the implementation of azcollection become hard to maintain, with very unfortunate requirements+dependencies as in https://github.com/ansible-collections/azure/blob/dev/plugins/module_utils/azure_rm_common.py#L228-L284

Unfortunate architecture (due historical reasons):

azure_rm_storageaccount  ...  ... azure_rm_postgresqlserver
        |                                   |    
        +-----------+            +----------+
                    |            |
                 +--v------------v---+
                 | azure_rm_comon.py |                       
                 +--+------------+---+
                    |            |
        +-----------+            +----------+
        |                                   |    
        v                                   v
azure.mgmt.storage ... ... ... azure.mgmt.rdbms.postgresql

@l3ender so you could do two things:

• refactor ansible module by ansible module, so only single modules depend on particular library of Azure SDK, like in the future only azure_rm_postgresserver requires azure.mgmt.rdbms.postgresql
• look at the current set of azure-cli requirements https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Linux.txt and upgrade every ansible module to use the same (more recent) azure-mgmt- version as azure-cli

Easier to maintain (better) architecture:

azure_rm_storageaccount  ...  ... azure_rm_postgresqlserver
        |                                   |    
        |                                   |    
        v                                   v
azure.mgmt.storage ... ... ... azure.mgmt.rdbms.postgresql

[Ramblurr]

Can someone post a workaround? How is anyone using this ansible collection on a modern linux/mac os system where they also have the latest azure cli installed for non-ansible tasks?

[Fred-sun]

@Ramblurr Related compute and Network resources have been submitted for PR upgrade to the latest, and other resources are under modification. Thank you very much!

[scottharwell]

@Ramblurr I'd recommend using the CLI on your host machine and using an Ansible execution environment with the Azure collection to isolate the two. Or, you could containerize both to keep them isolated and prevent dependency intersection. Use ansible-navigator to run your playbooks in a modern AAP 2+ way.

ansible-navigator run your_playbook.yml -i hosts \
--pae false \
--mode stdout \
--eei quay.io/scottharwell/azure-execution-env:latest \
--eev $HOME/.azure:/home/runner/.azure

[l3ender]

azure-cli-core was updated to 2.34.0 in https://github.com/ansible-collections/azure/pull/775; does that resolve this issue? It was released in v1.12.0.

[krapgras]

As this is unfortunately not yet fixed and most likely always be a challenge with the different development speeds i will share my fix.

Because i have been struggling on this as well and everyone talks about virtenv but nobody really shows how to use it.

What I did is 3 step, I created 2 tasks handling the azure-cli (creation + destruction) and then your main task(s) where you can use the Azure cli commands.

Role "install_azure-cli_venv"

---
# tasks file for roles/install_azure-cli_venv
- name: "Install azure-cli in {{ working_directory }}/venv"
  ansible.builtin.pip:
    name: azure-cli
    virtualenv: "/tmp/{{ inventory_hostname }}/venv"
    virtualenv_python: python3
    state: latest

Role 'main task'

---
# tasks file for roles/Run_AZ_Command
- name: Run AZ Command
  ansible.builtin.shell: |
    set -e
    source venv/bin/activate
      az command goes here                
  args:
    executable: /bin/bash
    chdir: "/tmp/{{ inventory_hostname }}"

Role 'remove_azure-cli_venv'

---
# tasks file for roles/remove_azure-cli_venv
- name: Remove python virtual env
  ansible.builtin.file:
    path: "/tmp/{{ inventory_hostname }}/venv"
    state: absent

I hope this helps someone struggling with the same :) I also agree that this is every time some extra space/data download but it's not the worst as you will only get the basic python3 setup and you will only download azure-cli with deps.

[Fred-sun]

@geekq @l3ender @howardjones @Ramblurr The dependencies in azure-requirements.txt have been updated, but they still conflict with the azure-cli after installation. There is no effective way to avoid such problems for the time being! If you can, only have it run in two separate virtual environments. Thanks!

[Fred-sun]

All relate dependencies has upgrade, It will contain in version v1.15.0, I will push this to release! So I will close this! Thank you very much!

[Fred-sun]

@howardjones Since there will be conflict between azure-cli and requirements-azure.txt when installed together, it is recommended to directly install the specified version of azure-cli to effectively avoid this problem, thank you!

Like this:
1. sudo pip3 install ansible
2. ansible-galaxy collection install azure.azcollection
3. pip3 install azure-cli==2.34.0
4. pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

Or: 
1. sudo pip3 install ansible
2. ansible-galaxy collection install azure.azcollection
3. pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
4. pip3 install azure-cli==2.34.0

[delahondes]

Dear all,

I've tried the latest solution and only the first version works for me: installing first azure.azcollection (I even have to specify pip3 install azure-mgmt-core==1.3.2 the very first time I install the collection otherwise pip is unable to find a solution, then only pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt works). Then finaly I install azure-cli with pip3 install azure-cli==2.34.0. This is done using Ubuntu 20.04 at the time of this writing.

In my relatively long experience of pip, I have found that pip is an unstable environment (probably due to its open nature) and results are hard to reproduce and tend to change over time...

In the second proposed solution, azure-cli works but azure.azcollection is broken, here is the error message I get while trying to create a new resource group:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' (/usr/local/lib/python3.8/dist-packages/azure/mgmt/recoveryservicesbackup/__init__.py)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on alpha's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

In this second solution, the pip3 install azure-cli==2.34.0 makes numerous changes in different libs that the azure collection does not seem to accommodate with.

[SpikePy]

i am having the same problem.