dell / dellemc-openmanage-ansible-modules

Dell OpenManage Ansible Modules
GNU General Public License v3.0
336 stars 163 forks source link

HTTP request fails for all ansible modules to OME #312

Closed liamwh closed 3 years ago

liamwh commented 3 years ago
Summary

HTTP request fails for all ansible modules to OME on version 3.7.0 build 82 from Ansible version 2.10.11 using versions 4.0.0 and 3.5.0 of dellemc.openmanage.

All ansible modules were working when OME was using version 3.5.0. Stopped working after upgrade to 3.7.0.

Component Name

dellemc.openmanage.ome_template_info: Using version 4.0.0

Ansible Version
ansible 2.10.11
  config file = /mnt/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Mar 18 2021, 08:58:41) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]
iDRAC or OpenManage Enterprise version

OpenManage Enterprise version Version 3.7.0 (Build 82)

Steps to Reproduce
- name: "Get OME template info"
  dellemc.openmanage.ome_template_info:
    hostname: "{{ openmanage_address }}"
    username: "{{ openmanage_username }}"
    password: "{{ openmanage_password }}"
  register: openmanage_template_info

- name: Debug
   debug:
     msg: "{{ openmanage_template_info }}"
Expected Results

Returned variable information for variable "openmanage_template_info".

Actual Results
<inventory_hostname> ESTABLISH LOCAL CONNECTION FOR USER: root
<inventory_hostname> EXEC /bin/sh -c 'echo ~root && sleep 0'
<inventory_hostname> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724 `" && echo ansible-tmp-1630673959.4661567-1071-232008919939724="` echo /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724 `" ) && sleep 0'
Using module file /mnt/ansible/collections/ansible_collections/dellemc/openmanage/plugins/modules/ome_device_info.py
<inventory_hostname> PUT /root/.ansible/tmp/ansible-local-986sy9x2939/tmpguwkeupi TO /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py
<inventory_hostname> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/ /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py && sleep 0'
<inventory_hostname> EXEC /bin/sh -c 'python3 /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py && sleep 0'
<inventory_hostname> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py", line 102, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.dellemc.openmanage.plugins.modules.ome_device_info', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib64/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/modules/ome_device_info.py", line 426, in <module>
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/modules/ome_device_info.py", line 384, in main
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/module_utils/ome.py", line 176, in __enter__
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/module_utils/ome.py", line 163, in invoke_request
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py", line 1399, in open_url
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py", line 1304, in open
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py", line 483, in https_open
  File "/usr/lib64/python3.6/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1361, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib64/python3.6/ssl.py", line 833, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib64/python3.6/ssl.py", line 590, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
fatal: [inventory_hostname]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1630673959.4661567-1071-232008919939724/AnsiballZ_ome_device_info.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.dellemc.openmanage.plugins.modules.ome_device_info', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/modules/ome_device_info.py\", line 426, in <module>\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/modules/ome_device_info.py\", line 384, in main\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/module_utils/ome.py\", line 176, in __enter__\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible_collections/dellemc/openmanage/plugins/module_utils/ome.py\", line 163, in invoke_request\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py\", line 1399, in open_url\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py\", line 1304, in open\n  File \"/usr/lib64/python3.6/urllib/request.py\", line 223, in urlopen\n    return opener.open(url, data, timeout)\n  File \"/usr/lib64/python3.6/urllib/request.py\", line 526, in open\n    response = self._open(req, data)\n  File \"/usr/lib64/python3.6/urllib/request.py\", line 544, in _open\n    '_open', req)\n  File \"/usr/lib64/python3.6/urllib/request.py\", line 504, in _call_chain\n    result = func(*args)\n  File \"/tmp/ansible_dellemc.openmanage.ome_device_info_payload_z8dzsetq/ansible_dellemc.openmanage.ome_device_info_payload.zip/ansible/module_utils/urls.py\", line 483, in https_open\n  File \"/usr/lib64/python3.6/urllib/request.py\", line 1352, in do_open\n    r = h.getresponse()\n  File \"/usr/lib64/python3.6/http/client.py\", line 1361, in getresponse\n    response.begin()\n  File \"/usr/lib64/python3.6/http/client.py\", line 311, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/lib64/python3.6/http/client.py\", line 272, in _read_status\n    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n  File \"/usr/lib64/python3.6/socket.py\", line 586, in readinto\n    return self._sock.recv_into(b)\n  File \"/usr/lib64/python3.6/ssl.py\", line 971, in recv_into\n    return self.read(nbytes, buffer)\n  File \"/usr/lib64/python3.6/ssl.py\", line 833, in read\n    return self._sslobj.read(len, buffer)\n  File \"/usr/lib64/python3.6/ssl.py\", line 590, in read\n    v = self._sslobj.read(len, buffer)\nsocket.timeout: The read operation timed out\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
[root@375c9d41528c ansible]# curl -v https://x.x.x.x:443/app/html/index.html -v
*   Trying x.x.x.x...
* TCP_NODELAY set
* Connected to x.x.x.x (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OBFUSCATED
*  start date: Apr  9 08:30:56 2021 GMT
*  expire date: May  9 00:00:00 2022 GMT
*  subjectAltName: host "x.x.x.x" matched cert's "x.x.x.x"
*  issuer: OBFUSCATED
*  SSL certificate verify ok.
> GET /app/html/index.html HTTP/1.1
> Host: x.x.x.x
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 06 Sep 2021 08:43:58 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Security-Policy: default-src 'self' 'unsafe-eval'; connect-src *; style-src 'self' 'unsafe-inline' blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' blob: data:
< Access-Control-Allow-Origin: *
< Last-Modified: Fri, 23 Jul 2021 00:24:09 GMT
< ETag: "afb-5c7bf6c0c9c40"
< Accept-Ranges: bytes
< Content-Length: 2811
< Vary: Accept-Encoding
< Content-Type: text/html; charset=UTF-8
<

Community Note

anupamaloke commented 3 years ago

@liamwh, are you experiencing this timeout issue with OME AD accounts or with a OME local account as well?

liamwh commented 3 years ago

We've experienced this issue using AD authentication, however we've identified the issue only occurs when running our Ansible Docker image (RedHat ubi8) on OSX, the issue does not occur when using the same docker image running on Windows... so it's most likely not a direct issue with OpenManage/the Ansible collection itself.

However, it is strange that all other ansible collections & modules are working completely fine from this image on OSX and it's only the OpenManage collection that has an issue... 🤔

Not sure how you want to proceed with this one, but if we can confirm that the Ansible code will run successfully from our pipeline agents then we can move on and workaround the issue.

anupamaloke commented 3 years ago

@liamwh, that is indeed interesting. Any chance you could open a support request with us so that the support team can investigate and collect diagnostics logs to help troubleshoot the issue?

However, it is strange that all other ansible collections & modules are working completely fine from this image on OSX and it's only the OpenManage collection that has an issue... 🤔

Are other ansible collections and modules that you are using from this image on OSX use REST APIs (i.e. run locally on the Ansible control node) or use SSH to login and execute tasks remotely on target nodes?

liamwh commented 3 years ago

Everything via API, no SSH. Also communicating to vCenter, iDRAC, iLO, OneView etc all totally fine.

I'm a bit busy for a support request at the moment but if you send me instructions on how to collect the logs I'll be happy to share them with you.

anupamaloke commented 3 years ago

@liamwh, thanks for sharing that information. So from what you just described, this issue manifests only when authenticating to OME using an AD account using Ansible modules running in a RHEL UBI container on OSX. I checked internally and they haven't seen this issue with OME Ansible modules.

I am not sure about what logs you need to collect on OME, but one test you could do is to check whether you are able to create a session for the AD user using ansible.builtin.uri module and OME REST API. You will want to run this playbook on the docker container on your OSX box. If this also times out, then it could be an issue with the network latency though I am not sure.

ome_user.yml

---
- hosts: ome
  connection: local
  gather_facts: no

  vars:
    ansible_python_interpreter: "/usr/bin/env python"

  tasks:
  - name: create user session
    ansible.builtin.uri:
      url: "https://{{ inventory_hostname }}/api/SessionService/Sessions"
      user: "{{ ome_ad_user }}"
      password: "{{ ome_ad_password }}"
      method: "POST"
      force_basic_auth: yes
      validate_certs: no
      headers:
        Accept: "application/json"
        Content-Type: "application/json"
      body:
        UserName: "{{ ome_ad_user }}"
        Password: "{{ ome_ad_password }}"
        SessionType: "API"
      body_format: json
      status_code: [200, 201]
    delegate_to: localhost
    register: result
$ ansible-playbook -vvv ome_user.yml -i hosts

ok: [192.168.10.10] => {
    "access_control_allow_origin": "*",
    "changed": false,
    "connection": "close",
    "content_length": "298",
    "content_security_policy": "default-src 'self' 'unsafe-eval'; connect-src *; style-src 'self' 'unsafe-inline' blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' blob: data:",
    "content_type": "application/json; odata.metadata=minimal",
    "cookies": {},
    "cookies_string": "",
    "date": "Thu, 09 Sep 2021 11:07:51 GMT",
    "elapsed": 1,
    "invocation": {
        "module_args": {
            "attributes": null,
            "body": {
                "Password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
                "SessionType": "API",
                "UserName": "<ome_ad_user>"
            },
            "body_format": "json",
            "client_cert": null,
            "client_key": null,
            "creates": null,
            "dest": null,
            "follow_redirects": "safe",
            "force": false,
            "force_basic_auth": true,
            "group": null,
            "headers": {
                "Accept": "application/json",
                "Content-Type": "application/json"
            },
            "http_agent": "ansible-httpget",
            "method": "POST",
            "mode": null,
            "owner": null,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "remote_src": false,
            "removes": null,
            "return_content": false,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": null,
            "status_code": [
                200,
                201
            ],
            "timeout": 30,
            "unix_socket": null,
            "unsafe_writes": false,
            "url": "https://192.168.10.10/api/SessionService/Sessions",
            "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "url_username": "<ome_ad_user>",
            "use_proxy": true,
            "user": "admin",
            "validate_certs": false
        }
    },
    "json": {
        "Description": "admin",
        "DirectoryGroup": [],
        "Id": "27b34b05-663f-471c-a355-828dbc4d0c1b",
        "IpAddress": "192.168.20.20",
        "LastAccessedTimeStamp": "2021-09-09 11:07:51.646",
        "Name": "API",
        "Password": null,
        "Roles": [
            "ADMINISTRATOR"
        ],
        "StartTimeStamp": "2021-09-09 11:07:51.646",
        "UserId": 10052,
        "UserName": "<ome_ad_user>"
    },
    "location": "https://192.168.10.10/api/SessionService/Sessions('27b34b05-663f-471c-a355-828dbc4d0c1b')",
    "msg": "OK (298 bytes)",
    "odata_version": "4.0",
    "redirected": false,
    "server": "Apache",
    "set_cookie": "rememberMe=deleteMe; Path=/api; Max-Age=0; Expires=Wed, 08-Sep-2021 11:07:51 GMT; SameSite=lax, JSESSIONID=deleteMe; Path=/api; Max-Age=0; Expires=Wed, 08-Sep-2021 11:07:51 GMT; SameSite=lax, rememberMe=deleteMe; Path=/api; Max-Age=0; Expires=Wed, 08-Sep-2021 11:07:52 GMT; SameSite=lax",
    "status": 201,
    "strict_transport_security": "max-age=31536000; includeSubDomains",
    "url": "https://100.69.127.50/api/SessionService/Sessions",
    "x_auth_token": "957ddafb-05c6-4dc0-bf62-921446c933f4",
    "x_content_type_options": "nosniff",
    "x_frame_options": "DENY",
    "x_xss_protection": "1; mode=block"
}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
192.168.10.10              : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
liamwh commented 3 years ago

@anupamaloke, thanks so much for your support. I was able to validate it was actually an invalid username/password configuration and bad luck that myself and another engineer made the same mistake. I apologise very much for wasting your time, and would advise including a bad auth response from OME if the username and password is incorrect!