Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)

ericcames commented 9 months ago

SUMMARY

Trying to use this module azure.azcollection.azure_rm_account_info with ansible automation platform 2.3 and we are getting the failure list in the title

ISSUE TYPE

Bug Report

COMPONENT NAME

azure.azcollection.azure_rm_account_info

ANSIBLE VERSION

[awx@aap ~]$ ansible --version
ansible [core 2.14.11]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /var/lib/awx/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.16 (main, Sep 22 2023, 17:57:55) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

azure.azcollection
version: 1.18.1
and
version: latest

CONFIGURATION

[defaults]
action_warnings = false
callbacks_enabled = profile_tasks
show_custom_stats = true
# inject_facts_as_vars = no
deprecation_warnings = false
forks = 10
gathering = explicit
roles_path = ./roles/
collections_path = ./collections/

[privilege_escalation]
become = false
become_method = sudo
become_user = root

[persistent_connection]
command_timeout = 30

OS / ENVIRONMENT

azure subscription account AAP 2.3 on RHEL 8

STEPS TO REPRODUCE

Set up a Microsoft Azure Resource Manager credential in AAP Create a template to run the playbook listed below Use this execution environment: quay.io/seanlee/ee-azure:1.12.0

---
- name: Get account info from Azure
  hosts: localhost
  connection: local

  tasks:

    - name: Get facts for current logged in user
      azure.azcollection.azure_rm_account_info:
...

EXPECTED RESULTS

I expect the playbook to pull back the account info.

ACTUAL RESULTS

{
  "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_azure.azcollection.azure_rm_account_info_payload_lev0wrek/ansible_azure.azcollection.azure_rm_account_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 256, in <module>\n    from azure.storage.blob import BlobServiceClient\nImportError: cannot import name 'BlobServiceClient' from 'azure.storage.blob' (/usr/local/lib/python3.8/site-packages/azure/storage/blob/__init__.py)\n",
  "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on aee9dfd4ba17's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter",
  "invocation": {
    "module_args": {
      "auth_source": "auto",
      "cloud_environment": "AzureCloud",
      "api_profile": "latest",
      "profile": null,
      "subscription_id": null,
      "client_id": null,
      "secret": null,
      "tenant": null,
      "ad_user": null,
      "password": null,
      "cert_validation_mode": null,
      "adfs_authority_url": null,
      "log_mode": null,
      "log_path": null,
      "x509_certificate_path": null,
      "thumbprint": null
    }
  },
  "_ansible_no_log": false,
  "changed": false
}

ericcames commented 9 months ago

@thisguyshouldworkforus I created an issue for what we are seeing.

Fred-sun commented 9 months ago

@ericcames After you installed aze.azcollection, did you install the related dependency files? Reinstall all dependent files and try again. Thank you!

ericcames commented 9 months ago

@ericcames After you installed aze.azcollection, did you install the related dependency files? Reinstall all dependent files and try again. Thank you!

@Fred-sun thank you for the feedback. When I read the dependencies section of the documentation is says there are none. I am installing the collections via the collections directory and a requirements file.

Not sure if you are redhat employee or not if you are you can find me via eric.ames@redhat.com. I am working with a customer to prove out that this collection works in their environment. We are looking for ideas on how to get this collection working. Thanks for your feedback!

A-Beck commented 9 months ago

I think the console is not loading dependencies correctly. You can see the python dependencies in the repo for the collection: https://github.com/ansible-collections/azure/blob/dev/requirements-azure.txt

Since you're using AAP, you should have access to the supported EE that includes these dependencies, IIRC

skertz commented 9 months ago

Tried the requirements-azure.txt with python3.6 and 3.7 both report "msgraph-sdk==1.0.0 not found". Using python3.8 and python 3.11 gets through this. However, running a playbook reports errors, ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'. Tried on fresh install of rh7 and rh8 machines.

ericcames commented 9 months ago

@alexander-aps tagging

Fred-sun commented 9 months ago

Tried the requirements-azure.txt with python3.6 and 3.7 both report "msgraph-sdk==1.0.0 not found". Using python3.8 and python 3.11 gets through this. However, running a playbook reports errors, ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'. Tried on fresh install of rh7 and rh8 machines.

@skertz This error is strange because we did not ask to install 'azure-storage'. Could you please provide your detailed execution environment and error information? Thank you!

Fred-sun commented 9 months ago

@ericcames Can you provide 'pip3 freeze' information and python version information? Thank you!

ericcames commented 9 months ago

@Fred-sun are you wanting this from the container or the controller?? This is the container we are using registry.redhat.io/ansible-automation-platform-23/ee-29-rhel8:latest

ericcames commented 9 months ago

@Fred-sun we created quay.io/zigfreed/ee-azure:latest and have validated that it works as expected with three other asure collections. ---

name: Get info Azure Resource Group hosts: localhost connection: local

tasks:
- name: Get all resource Groups info register: resourcegroupinfo azure.azcollection.azure_rm_resourcegroup_info:
- name: Print resourcegroupinfo ansible.builtin.debug: var: resourcegroupinfo ...

name: Get info Azure subscription_info hosts: localhost connection: local

tasks:
- name: Get facts for all subscriptions, including ones that are disabled. register: subscription_info azure.azcollection.azure_rm_subscription_info: all: true
- name: Print subscription_info ansible.builtin.debug: var: subscription_info ...

name: Create Azure Resource Group hosts: localhost connection: local vars: az_resource_group_name: 'woopdidoo' location: 'westus' customer: 'mickey.mouse'

tasks:
- name: Create a resource group azure.azcollection.azure_rm_resourcegroup: name: "{{ az_resource_group_name }}" location: "{{ location }}" state: present tags: created: ansbile delete: never demo: "{{ customer }}" ...

ericcames commented 9 months ago

This one does not work: ---

name: Get account info from Azure hosts: localhost connection: local

tasks:
- name: Get facts for current logged in user azure.azcollection.azure_rm_account_info: ...

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: error: MainError(additional_data={}, code='BadRequest', details=None, inner_error=InnerError(additional_data={'date': DateTime(2024, 2, 27, 12, 7, 25, tzinfo=Timezone('UTC'))}, client_request_id='24c090c3-8b51-4d10-bba2-d93ba2af6560', date=None, odata_type=None, request_id='f5d97fd3-a0cd-4f9e-8e69-02c50dd1c1b5'), message='/me request is only valid with delegated authentication flow.', target=None) fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3.9"}, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/home/runner/.ansible/tmp/ansible-tmp-1709035643.134935-26-217118459765709/AnsiballZ_azure_rm_account_info.py\", line 107, in \n _ansiballz_main()\n File \"/home/runner/.ansible/tmp/ansible-tmp-1709035643.134935-26-217118459765709/AnsiballZ_azure_rm_account_info.py\", line 99, in _ansiballz_main\n …

skertz commented 9 months ago

@Fred-sun

Redhat 8

ansible [core 2.15.3] config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.11/site-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /bin/ansible python version = 3.11.5 (main, Sep 22 2023, 15:34:29) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11) jinja version = 3.1.2 libyaml = True

Install: 1) ansible-galaxy collection install azure.azcollection 2) python3 -m pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

(python3 is 3.6) ERROR: Could not find a version that satisfies the requirement msgraph-sdk==1.0.0 (from versions: 1.0.0a0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0a4, 1.0.0a7, 1.0.0a8, 1.0.0a9, 1.0.0a10, 1.0.0a11, 1.0.0a12, 1.0.0a13) ERROR: No matching distribution found for msgraph-sdk==1.0.0

3) python3.11 -m pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

SUCCEEDS, but then:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount' fatal: [pd-ansible-3.production.iknowmed.com]: FAILED! => changed=false msg: Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on pd-ansible-3's Python /usr/bin/python3.11. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter

Isn't azure-mgmt-storage and azure-storage-blob defined in the requirements?

Fred-sun commented 9 months ago

@skertz There are definitions, all of our requirements files are in ~/ requirements-aze.txt. Thank you!


packaging
requests[security]
xmltodict
msgraph-sdk==1.0.0
azure-cli-core==2.34.0
azure-common==1.1.11
azure-identity==1.14.0
azure-mgmt-authorization==2.0.0
azure-mgmt-apimanagement==3.0.0
azure-mgmt-batch==16.2.0
azure-mgmt-cdn==11.0.0
azure-mgmt-compute==26.1.0
azure-mgmt-containerinstance==9.0.0
azure-mgmt-core==1.3.0
azure-mgmt-containerregistry==9.1.0
azure-containerregistry==1.1.0
azure-mgmt-containerservice==20.0.0
azure-mgmt-datalake-store==1.0.0
azure-mgmt-datafactory==2.0.0
azure-mgmt-dns==8.0.0
azure-mgmt-marketplaceordering==1.1.0
azure-mgmt-monitor==3.0.0
azure-mgmt-managedservices==6.0.0
azure-mgmt-managementgroups==1.0.0
azure-mgmt-network==19.1.0
azure-mgmt-nspkg==2.0.0
azure-mgmt-privatedns==1.0.0
azure-mgmt-redis==13.0.0
azure-mgmt-resource==21.1.0
azure-mgmt-rdbms==10.0.0
azure-mgmt-search==8.0.0
azure-mgmt-servicebus==7.1.0
azure-mgmt-sql==3.0.1
azure-mgmt-storage==19.0.0
azure-mgmt-trafficmanager==1.0.0b1
azure-mgmt-web==6.1.0
azure-nspkg==2.0.0
azure-storage-blob==12.11.0
azure-core==1.28.0
azure-keyvault==4.2.0
azure-mgmt-keyvault==10.0.0
azure-mgmt-cosmosdb==6.4.0
azure-mgmt-hdinsight==9.0.0
azure-mgmt-devtestlabs==9.0.0
azure-mgmt-loganalytics==12.0.0
azure-mgmt-automation==1.0.0
azure-mgmt-iothub==2.2.0
azure-iot-hub==2.6.1
azure-mgmt-recoveryservices==2.0.0
azure-mgmt-recoveryservicesbackup==3.0.0
azure-mgmt-notificationhubs==7.0.0
azure-mgmt-eventhub==10.1.0

ericcames commented 8 months ago

@Fred-sun we built these into the execution environment we are using for testing.

quay.io/zigfreed/ee-azure:latest

daviewales commented 8 months ago

I'm getting this exact error trying to use azure.azcollection.azure_rm_keyvaultsecret_info.

I have installed using pipx as follows:

pipx install --include-deps ansible
pipx runpip ansible install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt

My ansible configuration:

❯ ansible --version
ansible [core 2.16.4]
  config file = /home/dwales/.ansible.cfg
  configured module search path = ['/home/dwales/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dwales/.local/pipx/venvs/ansible/lib/python3.10/site-packages/ansible
  ansible collection location = /home/dwales/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/dwales/.local/bin/ansible
  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/home/dwales/.local/pipx/venvs/ansible/bin/python)
  jinja version = 3.1.3
  libyaml = True

Error message:

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_azure.azcollection.azure_rm_keyvaultsecret_info_payload_x0pvcnz2/ansible_azure.azcollection.azure_rm_keyvaultsecret_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 240, in <module>
    from azure.mgmt.core.tools import parse_resource_id, resource_id, is_valid_resource_id
ModuleNotFoundError: No module named 'azure'
fatal: [1.2.3.4]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "auth_source": "auto",
            "cert_validation_mode": null,
            "client_id": null,
            "cloud_environment": "AzureCloud",
            "log_mode": null,
            "log_path": null,
            "name": "secret",
            "password": null,
            "profile": null,
            "secret": null,
            "show_deleted_secret": false,
            "subscription_id": null,
            "tags": null,
            "tenant": null,
            "thumbprint": null,
            "vault_uri": "https://example.vault.azure.net",
            "version": "current",
            "x509_certificate_path": null
        }
    },
    "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on remote-vm's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"
}

I think the problem is that it's trying to import from azure.mgmt.core.tools on the remote server, rather than the local machine where I have installed the requirements.txt. If I add delegate_to: 127.0.0.1, it works perfectly.

Fred-sun commented 8 months ago

@ericcames Sorry, there is an issue that has not been clarified, graphrbac migrated to msgraph, supported python version >=3.8. Please use aze.azcollection >=2.0.0 to run only in an environment higher than python3.8. Thank you!

ericcames commented 8 months ago

@Fred-sun Thank you!

Fred-sun commented 7 months ago

@ericcames Are you trying the new version? Has the problem been solved?

nimbusparis commented 6 months ago

I got the same issue and after many try/error, I succeed after manually install python3, ansible on the target computer. Then, I also installed the azure collection and run pip3 install .../requirements/txt all manipulations had been done with root

Fred-sun commented 6 months ago

@nimbusparis What does your error message look like? Can you provide a detailed log? Thank you!

nimbusparis commented 6 months ago

@nimbusparis What does your error message look like? Can you provide a detailed log? Thank you!

Well, now, I don't have the issue anymore, I can try to uninstall the collection and python module, but I hope it wouldn't break anything! 😆 Also, I can try to install the required collection/python module on the fly in my playbook. In fact, I got the exact same error log as @ericcames at the begining of this thread.

nimbusparis commented 6 months ago

Here the vvvv log, hops it could help.

<51.255.204.38> ESTABLISH SSH CONNECTION FOR USER: user
<51.255.204.38> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' 51.255.204.38 '/bin/sh -c '"'"'echo ~user && sleep 0'"'"''
<51.255.204.38> (0, b'/home/user\n', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<51.255.204.38> ESTABLISH SSH CONNECTION FOR USER: user
<51.255.204.38> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' 51.255.204.38 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822 `" && echo ansible-tmp-1715760217.3001661-69163-31083769795822="` echo /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822 `" ) && sleep 0'"'"''
<51.255.204.38> (0, b'ansible-tmp-1715760217.3001661-69163-31083769795822=/home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822\n', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /usr/lib/python3/dist-packages/ansible_collections/azure/azcollection/plugins/modules/azure_rm_storageblob.py
<51.255.204.38> PUT /root/.ansible/tmp/ansible-local-69127aj5xlmin/tmp1a6bemrd TO /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py
<51.255.204.38> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' '[51.255.204.38]'
<51.255.204.38> (0, b'sftp> put /root/.ansible/tmp/ansible-local-69127aj5xlmin/tmp1a6bemrd /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py\n', b'OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/root/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/root/.ssh/known_hosts2\'\r\ndebug1: auto-mux: Trying existing master at \'/root/.ansible/cp/9afa1bc5be\'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug2: Server supports extension "limits@openssh.com" revision 1\r\ndebug2: Server supports extension "expand-path@openssh.com" revision 1\r\ndebug3: Sent message limits@openssh.com I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug3: server upload/download buffer sizes 261120 / 261120; using 261120 / 261120\r\ndebug3: server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/user\r\ndebug3: Looking up /root/.ansible/tmp/ansible-local-69127aj5xlmin/tmp1a6bemrd\r\ndebug2: Sending SSH2_FXP_STAT "/home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: sftp_upload: upload local "/root/.ansible/tmp/ansible-local-69127aj5xlmin/tmp1a6bemrd" to remote "/home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:154044\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 154044 bytes at 0\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<51.255.204.38> ESTABLISH SSH CONNECTION FOR USER: user
<51.255.204.38> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' 51.255.204.38 '/bin/sh -c '"'"'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/ /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py && sleep 0'"'"''
<51.255.204.38> (0, b'', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<51.255.204.38> ESTABLISH SSH CONNECTION FOR USER: user
<51.255.204.38> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' -tt 51.255.204.38 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=mnwrxfhewwhoddyzirtxjtldezxfwwhl] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-mnwrxfhewwhoddyzirtxjtldezxfwwhl ; /usr/bin/python3 /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/AnsiballZ_azure_rm_storageblob.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<51.255.204.38> (1, b'\r\n\r\n{"exception": "Traceback (most recent call last):\\n  File \\"/tmp/ansible_azure.azcollection.azure_rm_storageblob_payload_1q1m5w5f/ansible_azure.azcollection.azure_rm_storageblob_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\\", line 223, in <module>\\n    from msrestazure.azure_active_directory import AADTokenCredentials\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrestazure/__init__.py\\", line 28, in <module>\\n    from .azure_configuration import AzureConfiguration\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrestazure/azure_configuration.py\\", line 34, in <module>\\n    from msrest import Configuration\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrest/__init__.py\\", line 28, in <module>\\n    from .configuration import Configuration\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrest/configuration.py\\", line 38, in <module>\\n    from .universal_http.requests import (\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrest/universal_http/__init__.py\\", line 53, in <module>\\n    from ..exceptions import ClientRequestError, raise_with_traceback\\n  File \\"/usr/local/lib/python3.10/dist-packages/msrest/exceptions.py\\", line 31, in <module>\\n    from azure.core.exceptions import SerializationError, DeserializationError\\nModuleNotFoundError: No module named \'azure.core\'\\n", "failed": true, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on vps350593\'s Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter", "invocation": {"module_args": {"resource_group": "soft-rg", "storage_account_name": "softsa", "container": "artifacts", "blob": "soft-backend/soft-backend.zip", "dest": "/root/tmp/soft/backend.zip", "auth_source": "auto", "cloud_environment": "AzureCloud", "api_profile": "latest", "append_tags": true, "blob_type": "block", "force": false, "state": "present", "profile": null, "subscription_id": null, "client_id": null, "secret": null, "tenant": null, "ad_user": null, "password": null, "cert_validation_mode": null, "adfs_authority_url": null, "log_mode": null, "log_path": null, "x509_certificate_path": null, "thumbprint": null, "tags": null, "src": null, "batch_upload_src": null, "batch_upload_dst": null, "public_access": null, "content_type": null, "content_encoding": null, "content_language": null, "content_disposition": null, "cache_control": null, "content_md5": null}}}\r\n', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 51.255.204.38 closed.\r\n")
<51.255.204.38> Failed to connect to the host via ssh: OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 51.255.204.38 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 69017
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet_timeout: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to 51.255.204.38 closed.
<51.255.204.38> ESTABLISH SSH CONNECTION FOR USER: user
<51.255.204.38> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/9afa1bc5be"' 51.255.204.38 '/bin/sh -c '"'"'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1715760217.3001661-69163-31083769795822/ > /dev/null 2>&1 && sleep 0'"'"''
<51.255.204.38> (0, b'', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /root/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 51.255.204.38 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/root/.ansible/cp/9afa1bc5be'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69017\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_azure.azcollection.azure_rm_storageblob_payload_1q1m5w5f/ansible_azure.azcollection.azure_rm_storageblob_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 223, in <module>
    from msrestazure.azure_active_directory import AADTokenCredentials
  File "/usr/local/lib/python3.10/dist-packages/msrestazure/__init__.py", line 28, in <module>
    from .azure_configuration import AzureConfiguration
  File "/usr/local/lib/python3.10/dist-packages/msrestazure/azure_configuration.py", line 34, in <module>
    from msrest import Configuration
  File "/usr/local/lib/python3.10/dist-packages/msrest/__init__.py", line 28, in <module>
    from .configuration import Configuration
  File "/usr/local/lib/python3.10/dist-packages/msrest/configuration.py", line 38, in <module>
    from .universal_http.requests import (
  File "/usr/local/lib/python3.10/dist-packages/msrest/universal_http/__init__.py", line 53, in <module>
    from ..exceptions import ClientRequestError, raise_with_traceback
  File "/usr/local/lib/python3.10/dist-packages/msrest/exceptions.py", line 31, in <module>
    from azure.core.exceptions import SerializationError, DeserializationError
ModuleNotFoundError: No module named 'azure.core'
fatal: [51.255.204.38]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ad_user": null,
            "adfs_authority_url": null,
            "api_profile": "latest",
            "append_tags": true,
            "auth_source": "auto",
            "batch_upload_dst": null,
            "batch_upload_src": null,
            "blob": "soft-backend/soft-backend.zip",
            "blob_type": "block",
            "cache_control": null,
            "cert_validation_mode": null,
            "client_id": null,
            "cloud_environment": "AzureCloud",
            "container": "artifacts",
            "content_disposition": null,
            "content_encoding": null,
            "content_language": null,
            "content_md5": null,
            "content_type": null,
            "dest": "/root/tmp/soft/backend.zip",
            "force": false,
            "log_mode": null,
            "log_path": null,
            "password": null,
            "profile": null,
            "public_access": null,
            "resource_group": "soft-rg",
            "secret": null,
            "src": null,
            "state": "present",
            "storage_account_name": "softsa",
            "subscription_id": null,
            "tags": null,
            "tenant": null,
            "thumbprint": null,
            "x509_certificate_path": null
        }
    },
    "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on vps350593's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"
}

Fred-sun commented 6 months ago

@nimbusparis Can you provide your test script? I'd like to try, thank you very much!

nimbusparis commented 6 months ago

Inventory:

all:
  children:
    ovh:
      hosts: xxx.xxx.xxx.xxx

group_vars/ovh.yaml

servicePrincipal:
  appId: xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx
  secret: fdsdfdsfsdfsdfsdfsf
  tenant:  xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx

Playbook:

- name: Deployment
  remote_user: user
  become: True
  hosts: ovh

  tasks:
  - name: Install Python3
    ansible.builtin.raw: apt install python3

  - name: Login to Azure
    # Use Azure CLI (installed manually) to connect to Azure
    ansible.builtin.command: "az login --service-principal -u {{servicePrincipal.appId}} -p {{servicePrincipal.secret}} -t {{servicePrincipal.tenant}}" 

  - name: Download Pixhint packages
    azure.azcollection.azure_rm_storageblob:
      resource_group: soft-rg
      storage_account_name: softsa
      container: artifacts
      blob: "soft/soft-backend.zip"
      dest: "~/tmp/soft/soft-backend.zip"

ansible-collections / azure