search

ansible-collections / azure

Development area for Azure Collections
https://galaxy.ansible.com/azure/azcollection
GNU General Public License v3.0
247 stars 331 forks source link

ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount' #1496

Closed skertz closed 6 months ago

skertz commented 8 months ago
SUMMARY
ISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
ansible [core 2.14.6]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.5 (main, Sep 22 2023, 15:34:29) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/bin/python3)
  jinja version = 3.1.3
  libyaml = Tru
COLLECTION VERSION
Collection         Version
------------------ -------
azure.azcollection 2.2.0
CONFIGURATION
OS / ENVIRONMENT

Red Hat Enterprise Linux 8.9 (Ootpa)

STEPS TO REPRODUCE
yum install python311
sudo update-alternatives --config python3
python3 -m ensurepip
python3 -m pip install ansible
ansible-galaxy collection install azure.azcollection
pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

Playbook:
---
- hosts: azurevm
  become: true
  collections:
    - azure.azcollection
  vars:
    ansible_python_interpreter: /bin/python3
  tasks: 
    - name: Print message
      ansible.builtin.debug:
        msg: Hello from Ansible
    - name: Query all the resources in the resource group
      azure_rm_resource_info:
        resource_group: "rg-myRG"
        resource_type: resources
EXPECTED RESULTS
ACTUAL RESULTS
TASK [Query all the resources in the resource group] ********************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'
fatal: [dv-ansible-4]: FAILED! => changed=false 
  msg: Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on pd-ansible-4's Python /bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter
Fred-sun commented 8 months ago

@skertz The Azure-storage already abandoned (v0.37.0), but the error "azure. storage. cloudstorageaccount 'should no longer appear in the abandoned after package. Suspicion is due to incompatible reasons. Could you please provide some information about your working environment? (pip list, python version, System....) ? Thank you!

skertz commented 8 months ago

Red Hat Enterprise Linux 8.9 (Ootpa) deployed to Azure - clean build. I've deployed RH7 and RH8 and rebuilt different times.

There are 3 programs which provide 'python3'.

  Selection    Command
-----------------------------------------------
*  1           /usr/bin/python3.6
   2           /usr/bin/python3.8
 + 3           /usr/bin/python3.11

pip list

 Package                              Version
------------------------------------ -----------
adal                                 1.2.7
aiohttp                              3.9.3
aiosignal                            1.3.1
ansible                              9.3.0
ansible-core                         2.16.4
anyio                                4.3.0
applicationinsights                  0.11.10
argcomplete                          1.12.3
attrs                                23.2.0
azure-cli-core                       2.34.0
azure-cli-telemetry                  1.0.6
azure-common                         1.1.11
azure-containerregistry              1.1.0
azure-core                           1.28.0
azure-identity                       1.14.0
azure-iot-hub                        2.6.1
azure-keyvault                       4.2.0
azure-keyvault-certificates          4.7.0
azure-keyvault-keys                  4.8.0
azure-keyvault-secrets               4.7.0
azure-mgmt-apimanagement             3.0.0
azure-mgmt-authorization             2.0.0
azure-mgmt-automation                1.0.0
azure-mgmt-batch                     16.2.0
azure-mgmt-cdn                       11.0.0
azure-mgmt-compute                   26.1.0
azure-mgmt-containerinstance         9.0.0
azure-mgmt-containerregistry         9.1.0
azure-mgmt-containerservice          20.0.0
azure-mgmt-core                      1.3.0
azure-mgmt-cosmosdb                  6.4.0
azure-mgmt-datafactory               2.0.0
azure-mgmt-datalake-store            1.0.0
azure-mgmt-devtestlabs               9.0.0
azure-mgmt-dns                       8.0.0
azure-mgmt-eventhub                  10.1.0
azure-mgmt-hdinsight                 9.0.0
azure-mgmt-iothub                    2.2.0
azure-mgmt-keyvault                  10.0.0
azure-mgmt-loganalytics              12.0.0
azure-mgmt-managedservices           6.0.0
azure-mgmt-managementgroups          1.0.0
azure-mgmt-marketplaceordering       1.1.0
azure-mgmt-monitor                   3.0.0
azure-mgmt-network                   19.1.0
azure-mgmt-notificationhubs          7.0.0
azure-mgmt-nspkg                     2.0.0
azure-mgmt-privatedns                1.0.0
azure-mgmt-rdbms                     10.0.0
azure-mgmt-recoveryservices          2.0.0
azure-mgmt-recoveryservicesbackup    3.0.0
azure-mgmt-redis                     13.0.0
azure-mgmt-resource                  21.1.0
azure-mgmt-search                    8.0.0
azure-mgmt-servicebus                7.1.0
azure-mgmt-sql                       3.0.1
azure-mgmt-storage                   19.0.0
azure-mgmt-trafficmanager            1.0.0b1
azure-mgmt-web                       6.1.0
azure-nspkg                          2.0.0
azure-storage-blob                   12.11.0
bcrypt                               4.1.2
certifi                              2024.2.2
cffi                                 1.16.0
charset-normalizer                   3.3.2
cryptography                         42.0.5
Deprecated                           1.2.14
frozenlist                           1.4.1
h11                                  0.14.0
h2                                   4.1.0
hpack                                4.0.0
httpcore                             1.0.4
httpx                                0.27.0
humanfriendly                        10.0
hyperframe                           6.0.1
idna                                 3.6
importlib-metadata                   6.11.0
isodate                              0.6.1
Jinja2                               3.1.3
jmespath                             1.0.1
knack                                0.9.0
MarkupSafe                           2.1.5
microsoft-kiota-abstractions         1.3.1
microsoft-kiota-authentication-azure 1.0.0
microsoft-kiota-http                 1.3.1
microsoft-kiota-serialization-json   1.1.0
microsoft-kiota-serialization-text   1.0.0
msal                                 1.27.0
msal-extensions                      0.3.1
msgraph-core                         1.0.0
msgraph-sdk                          1.0.0
msrest                               0.7.1
msrestazure                          0.6.4
multidict                            6.0.5
oauthlib                             3.2.2
opentelemetry-api                    1.23.0
opentelemetry-sdk                    1.23.0
opentelemetry-semantic-conventions   0.44b0
packaging                            21.3
paramiko                             2.12.0
pendulum                             3.0.0
pip                                  24.0
pkginfo                              1.10.0
portalocker                          1.7.1
psutil                               5.9.8
pycparser                            2.21
Pygments                             2.17.2
PyJWT                                2.8.0
PyNaCl                               1.5.0
pyOpenSSL                            24.1.0
pyparsing                            3.1.2
PySocks                              1.7.1
python-dateutil                      2.9.0.post0
PyYAML                               6.0.1
requests                             2.31.0
requests-oauthlib                    1.4.0
resolvelib                           0.8.1
setuptools                           65.5.1
six                                  1.16.0
sniffio                              1.3.1
std-uritemplate                      0.0.55
tabulate                             0.9.0
time-machine                         2.14.0
typing_extensions                    4.10.0
tzdata                               2024.1
uamqp                                1.6.8
urllib3                              2.2.1
wrapt                                1.16.0
xmltodict                            0.13.0
yarl                                 1.9.4
zipp                                 3.18.1
Fred-sun commented 8 months ago

@skertz See you have three python interpretation versions, is it not in the execution, the python interpreter path is wrong, thank you!

skertz commented 8 months ago

Typically the path is Python /usr/libexec/platform-python However, I am setting 

  vars:
    ansible_python_interpreter: /usr/bin/python3
lrwxrwxrwx. 1 root root 25 Feb 18 16:54 /bin/python3 -> /etc/alternatives/python3
[root@pd-ansible-4 ~]# ll /etc/alternatives/python3
lrwxrwxrwx. 1 root root 19 Mar 18 17:07 /etc/alternatives/python3 -> /usr/bin/python3.11

When setting to Python 3.6 - I see I'm missing the packaging module as expected because I did not install with 3.6

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'packaging'
fatal:

It is when trying to use Python 3.8 and 3.11 I see the same error for azure.storage.cloudstorageaccount

If trying to use the default /usr/libexec/platform-python3.6 I can not even install the azure collection requirements. 

  Could not find a version that satisfies the requirement msgraph-sdk==1.0.0 (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 4)) (from versions: 1.0.0a0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0a4, 1.0.0a7, 1.0.0a8, 1.0.0a9, 1.0.0a10, 1.0.0a11, 1.0.0a12, 1.0.0a13)
No matching distribution found for msgraph-sdk==1.0.0 (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 4))
Fred-sun commented 8 months ago

@skertz I see you made a soft link to '/usr/bin/python3'. What version of python3 did you have before you built the link (the minimum python version 3.6 required by msgraph-sdk)? The module cannot be found. Is it because the interpreter of the soft link you specified does not match with the installed directory that it cannot work? Could you please help confirm these two points? Thank you!

skertz commented 8 months ago

The version of python originally is 3.6. This is a fresh azure vm deployment. However when using 3.6 - msgraphsdk cannot be installed from the azcollection requirements file due to missing version available.

[root@pd-ansible-4 ~]# python3 --version
Python 3.6.8
[root@pd-ansible-4 ~]# python3 -m pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Collecting packaging (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl
Requirement already satisfied: requests[security] in /usr/lib/python3.6/site-packages (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 2))
Collecting xmltodict (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/94/db/fd0326e331726f07ff7f40675cd86aa804bfd2e5016c727fa761c934990e/xmltodict-0.13.0-py2.py3-none-any.whl
Collecting msgraph-sdk==1.0.0 (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 4))
  Could not find a version that satisfies the requirement msgraph-sdk==1.0.0 (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 4)) (from versions: 1.0.0a0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0a4, 1.0.0a7, 1.0.0a8, 1.0.0a9, 1.0.0a10, 1.0.0a11, 1.0.0a12, 1.0.0a13)
No matching distribution found for msgraph-sdk==1.0.0 (from -r /root/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt (line 4))

Which is why I've tried Python 3.8, 3.9 , 3.11 , and different versions of the collections. All comes back to the same cloudstorageaccount error. My setup is simple, deploy a vm in azure. And then run these commands:

yum install python311
sudo update-alternatives --config python3
python3 -m ensurepip
python3 -m pip install ansible
ansible-galaxy collection install azure.azcollection
pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
Fred-sun commented 6 months ago

@skertz Yes, msgraph-sdk relies on a version of python higher than python 3.8. Could you please try it according to the following instructions, retry your playbook! thanks!


yum install python3.9
sudo update-alternatives --config python3
pip3 install ansible[azure]
ansible-galaxy collection install azure.azcollection
pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
skertz commented 6 months ago

@Fred-sun - yes the msgraph -sdk is needed by Python >= 3.9 - But as noted prior to this suggestion. I have tried Python 3.9 and 3.11. Everything all comes back to the error for cloudstorageaccount The error was: ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'

Fred-sun commented 6 months ago

@skertz Yes, I rebuilt a VM and ran the following command without any problems. thank you.


sudo update-alternatives --config python3
pip3 install ansible[azure]
ansible-galaxy collection install azure.azcollection
pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
skertz commented 6 months ago

I have done this on RedHat 8, RedHat9, CentOS 8.5 deployed fresh into Azure I have done this with Python3.9 and 3.10 and 3.11 All similiar results with the same error for cloudstorageaccount

Here is my history file on brand new vm in Azure 

    8  yum install python3.9
    9  sudo update-alternatives --config python3
   10  python3 -m ensurepip
   11  pip3 install ansible[azure]
   12  ansible-galaxy collection install azure.azcollection
   13  /usr/local/bin/ansible-galaxy collection install azure.azcollection
   14  /usr/local/bin/ansible-galaxy collection install azure.azcollection --force
   15  pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
   16  history

I run a playbook.

  become: true
  collections:
    - azure.azcollection
  tasks:
    - name: Print message
      ansible.builtin.debug:
        msg: Hello from Ansible
    # - name: Get Azure subscription
    - name: Query all the resources in the resource group
      azure_rm_resource_info:
        resource_group: "rg-myResourceGroup"
        resource_type: resources

Then this error 

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'
Fred-sun commented 6 months ago

@skertz Re-test on redhat 8 to obtain resource information. The following is the installation process. Thank you.

    1  python3
    2  cat /etc/ssh/sshd_config
    3  sudo cat /etc/ssh/sshd_config
    4  sudo yum update
    5  sudo  yum install python3.9
    6  sudo update-alternatives --config python3
    7  python3 -m ensurepip
    8  sudo pip3 install ansible[azure]
    9  ansible --version
   10  ansible-galaxy collection install azure.azcollection
   11  ansible-galaxy collection install azure.azcollection --force
   12  sudo pip3 install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt
   13  pip3 list

Playbook:
---
- name: For test
  hosts: localhost
  become: true
  collections:
    - azure.azcollection
  tasks:
    - name: Print message
      ansible.builtin.debug:
        msg: Hello from Ansible
    # - name: Get Azure subscription
    - name: Query all the resources in the resource group
      azure_rm_resource_info:
        resource_group: v-xisuRG02
        resource_type: resources
      register: output
    - debug:
        var: output
Return info:
TASK [Query all the resources in the resource group] ************************************************************
ok: [localhost]

TASK [debug] ****************************************************************************************************
ok: [localhost] => {
    "output": {
        "changed": false,
        "failed": false,
        "response": [
            {
                "id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/v-xisuRG02/providers/Microsoft.Network/virtualNetworks/sqlmi003",
                "location": "eastus",
                "name": "sqlmi003",
                "type": "Microsoft.Network/virtualNetworks"
            },
            {
          ..........
Fred-sun commented 6 months ago

@skertz Did you encounter any errors during the installation process?

Fred-sun commented 6 months ago

@skertz In addition, I see that the python version you specified is 3.11, if you directly specify the python version is 3.9, what is the result?

Fred-sun commented 6 months ago

@skertz I still need more information to clarify this issue, can you help to provide '-vvvv 'information to execute the script, as well as your entire operation process, thank you!

skertz commented 6 months ago

This is issue can be marked resolved. Working with colleague, found that the ansible automation controller was missing the collections, even though the server we were executing on had all the installation pieces. If using "delegate" it would work on the server where items installed. Strange that the error was for missing azure cloudstorageaccount module.