Possible conflict between azure-cli package and fence-agents-azure-arm package

imabedalghafer commented 2 years ago

This is autogenerated. Please review and update as needed.

Describe the bug

when installing the fence-agents-azure-arm package to configure azure fence agent on RHEL cluster as per the document: https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-rhel-pacemaker#cluster-installation and then trying to install the azure cli on the same machine, the azure cli commands is failing , a below is an example "az login".

the installtion of fence-agents-azure-arm package is installing the python3-azure-sdk package, but I am not sure how this would affect the use of the azure cli. if the python3-azure-sdk package is removed, the azure cli commands works fine.

note : trying the older versions of azure-cli would not help too. would you please help us to understand how they are correlated ? and if there is any solution to have both of them on the same machine ?

Command Name az login

Errors:

The command failed with an unexpected error. Here is the traceback:
__init__() got an unexpected keyword argument 'user_agent'
Traceback (most recent call last):
  File "/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/command_modules/profile/custom.py", line 158, in login
    use_cert_sn_issuer=use_cert_sn_issuer)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 175, in login
    subscriptions = subscription_finder.find_using_specific_tenant(tenant, credential)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 815, in find_using_specific_tenant
    client = self._create_subscription_client(credential)
  File "/lib64/az/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 839, in _create_subscription_client
    **client_kwargs)
TypeError: __init__() got an unexpected keyword argument 'user_agent'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. it can be tested also on centos 8 machines with HA repos enabled, and azure cli repos enabled.

Put any pre-requisite steps here...
yum install fence-agents-azure-arm
yum install azure-cli
az login --service-principal --allow-no-subscriptions -u {} -p {} -t {}

Expected Behavior

to be able to login successfully

Environment Summary

Linux-4.18.0-269.el8.x86_64-x86_64-with-centos-8, CentOS Stream 8
Python 3.6.8
Installer: RPM

azure-cli 2.31.0

Additional Context

yonzhan commented 2 years ago

@jiasli for awareness

jiasli commented 2 years ago

@imabedalghafer, I am not familiar with fence-agents-azure-arm, but I can't seem to install them:

docker run -it --rm centos

[root@8f0358034b9c /]# yum install python3-azure-sdk
Failed to set locale, defaulting to C.UTF-8
CentOS Linux 8 - AppStream                                                              179 kB/s | 8.4 MB     00:47
CentOS Linux 8 - BaseOS                                                                 743 kB/s | 3.6 MB     00:05
CentOS Linux 8 - Extras                                                                  12 kB/s |  10 kB     00:00
No match for argument: python3-azure-sdk
Error: Unable to find a match: python3-azure-sdk
...
[root@8f0358034b9c /]# yum install -y pcs pacemaker fence-agents-azure-arm nmap-ncat
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:25 ago on Fri Dec 24 02:50:10 2021.
No match for argument: pcs
No match for argument: pacemaker
No match for argument: fence-agents-azure-arm
Error: Unable to find a match: pcs pacemaker fence-agents-azure-arm

Do I need to add some repo first?

Azure CLI has an entry script /usr/bin/az:

# command -v az
/usr/bin/az

# cat /usr/bin/az
#!/usr/bin/env bash
bin_dir=`cd "$(dirname "$BASH_SOURCE[0]")"; pwd`
python_cmd=python3
if command -v python3.6 &>/dev/null; then python_cmd=python3.6; fi
AZ_INSTALLER=RPM PYTHONPATH="$bin_dir/../lib64/az/lib/python3.6/site-packages" $python_cmd -sm azure.cli "$@"

azure-mgmt-resource SDK is installed in a virtual environment. That's why the entry script /usr/bin/az has

PYTHONPATH="$bin_dir/../lib64/az/lib/python3.6/site-packages"

Perhaps you can share the result of

# PYTHONPATH="$bin_dir/../lib64/az/lib/python3.6/site-packages" python3.6 -m pip list -v
...
azure-mgmt-resource                     20.0.0    /usr/lib64/az/lib/python3.6/site-packages pip

imabedalghafer commented 2 years ago

@jiasli Thank you for your reply. I do apologize I forget to mention, that yes, you would need to enable the high availability repository of Centos , which can be found under:/etc/yum.repos.d/CentOS-Stream-HighAvailability.repo you would need to edit the file and enable the repository , or it can be done via below:

sed -i 's/enabled=0/enabled=1/g' /etc/yum.repos.d/CentOS-Stream-HighAvailability.repo

The output for the above command (the verbose did not return for me the full path for that package) :

# PYTHONPATH="$bin_dir/../lib64/az/lib/python3.6/site-packages" python3.6 -m pip list --format columns -vvv | grep azure-mgmt-resource
azure-mgmt-resource                              2.1.0
azure-mgmt-resourcegraph                         1.0.0

Please let me know if any further information is needed.

jiasli commented 2 years ago

For the provided result

The version of azure-mgmt-resource is 2.1.0 which is different from what azure-cli requires - 20.0.0. Also, the installation location is missing from the result of pip list command. We need that information to identify where azure-mgmt-resource 2.1.0 is installed.

It is possible PYTHONPATH is not set correctly, so the system pip is invoked. Could you try

# ls /lib64/az/lib/python3.6/site-packages

It should show azure-cli's lib.

Repro

There doesn't seem to be a CentOS Stream official image, so I tried the official centos image and also reproduced:

> docker run -it --rm centos

# sed -i 's/enabled=0/enabled=1/g' /etc/yum.repos.d/CentOS-Linux-HighAvailability.repo
# yum install -y pcs pacemaker fence-agents-azure-arm nmap-ncat

# rpm --import https://packages.microsoft.com/keys/microsoft.asc
# echo -e "[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc" | tee /etc/yum.repos.d/azure-cli.repo
# yum install azure-cli

# az login
TypeError: __init__() got an unexpected keyword argument 'user_agent'

Root cause

Because fence-agents-azure-arm installs azure-mgmt-resource 2.1.0 which is under a regular packages azure-mgmt due to the /usr/lib/python3.6/site-packages/azure/mgmt/__init__.py file, it takes precedence over azure-mgmt-resource 20.0.0 which is under a namespace package azure-mgmt installed at /usr/lib64/az/lib/python3.6/site-packages.

# PYTHONPATH="/lib64/az/lib/python3.6/site-packages" python3.6 -m pip list --format columns -vvv | grep azure-mgmt-resource
azure-mgmt-resource                              20.0.0      /usr/lib64/az/lib/python3.6/site-packages pip
azure-mgmt-resourcegraph                         1.0.0       /usr/lib/python3.6/site-packages

# PYTHONPATH="/lib64/az/lib/python3.6/site-packages" python3.6 -c "import azure.mgmt.resource; print(azure.mgmt.resource.__path__)"
['/usr/lib/python3.6/site-packages/azure/mgmt/resource']

# PYTHONPATH="/lib64/az/lib/python3.6/site-packages" python3.6 -c "import azure.mgmt; print(azure.mgmt)"
<module 'azure.mgmt' from '/usr/lib/python3.6/site-packages/azure/mgmt/__init__.py'>

# cat /usr/lib/python3.6/site-packages/azure/mgmt/resource/version.py
VERSION = "2.1.0"

As az is invoked using the system python, include-system-site-packages actually doesn't work here:

# cat /lib64/az/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.6.8

We've seen similar issue before: https://github.com/Azure/azure-cli/issues/19027

Possible solution

As a solution, we can consider using the python in the virtual env to launch azure.cli, so that include-system-site-packages is honored.
You may also consider reporting this issue to fence-agents-azure-arm since it is using a very old and deprecated azure-mgmt-resource 2.1.0.

Workaround

Current as a workaround, you may use pip to install Azure CLI to make it isolated from the system python lib:

python3 -m venv az-env
. az-env/bin/activate
pip install --upgrade pip
pip install azure-cli

imabedalghafer commented 2 years ago

@jiasli Thank you so much for your help and analysis.

Regarding the first solution :

As a solution, we can consider using the python in the virtual env to launch azure.cli, so that include-system-site-packages is honored.

would you please elaborate more about the solution (as far as I understood, it would be something similar to the workaround) ? Also regarding the workaround, would it be possible to update the az script as of this scenario to automatically activate the virtual environment and start using the az command from there ?

Regards, Ibrahim Abedalghafer

imabedalghafer commented 2 years ago

@jiasli What I meat for the update of the az command to activate the virtual environment in the az command is something like below:

# vi /bin/az
--
#!/usr/bin/env bash
#bin_dir=`cd "$(dirname "$BASH_SOURCE[0]")"; pwd`
#python_cmd=python3
#if command -v python3.6 &>/dev/null; then python_cmd=python3.6; fi
#AZ_INSTALLER=RPM PYTHONPATH="$bin_dir/../lib64/az/lib/python3.6/site-packages" $python_cmd -sm azure.cli "$@"
## Adding the below as a workaround for issue :  20820 ####
source /root/az-env/bin/activate
python -sm azure.cli "$@"

I have tested it and it is working , and just want to confirm if it can be considered as a valid workaround to use.

jiasli commented 2 years ago

would you please elaborate more about the solution (as far as I understood, it would be something similar to the workaround) ?

Ah. "We" here means us, Microsoft. If you check our DEB packages, it actually contains its own python. I am not really sure why the same is not done for the RPM package (it was developed by the previous team years ago), but I will definitely work on this in the future.

Actually, you don't need to activate the venv for az to work. You can simply launch az in the venv:

docker run -it --rm --workdir /root python bash

python -m venv az-env
az-env/bin/pip install azure-cli
az-env/bin/az --version

You may also add az-env/bin/ to PATH so that az is available everywhere:

export PATH="/root/az-env/bin:$PATH"
az --version

imabedalghafer commented 2 years ago

@jiasli Thank you so much for your help on the case. The tests now are working fine with the provided workaround. please feel free to close the issue or if you would like to keep it open for future reference it is up to you 😊

Azure / azure-cli