ERROR [io.undertow.request] (default task-7) UT005023: Exception handling request to /ovirt-engine/api/v4/clusters: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Problem following 'gluster_volumes' link in Clusters entity

[bit4man]

4.5.0 upgrade from 4.4 is working, but cannot upgrade the cluster definitions to 4.5. The ansible playbook /usr/share/ovirt-engine/ansible-runner-service-project/project/ovirt-cluster-upgrade.yml fails very early with

TASK [ovirt.ovirt.cluster_upgrade : Get cluster info] *******************************************************************************************************************************************************
task path: /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/cluster_upgrade/tasks/main.yml:39
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532 `" && echo ansible-tmp-1658173173.538555-28152-4648540958532="` echo /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532 `" ) && sleep 0'
Loading collection redhat.rhv from /usr/share/ansible/collections/ansible_collections/redhat/rhv
Using module file /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_cluster_info.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-28090gdjgm3ul/tmpfqjm3gjz TO /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532/AnsiballZ_ovirt_cluster_info.py
<127.0.0.1> EXEC /bin/sh -c 'chmod
[error.log](https://github.com/oVirt/ovirt-engine/files/9135395/error.log)
 u+x /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532/ /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532/AnsiballZ_ovirt_cluster_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.8 /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532/AnsiballZ_ovirt_cluster_info.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1658173173.538555-28152-4648540958532/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_ovirt_cluster_info_payload_n56eib2s/ansible_ovirt_cluster_info_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_cluster_info.py", line 112, in main
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/services.py", line 5747, in list
    return self._internal_get(headers, query, wait)
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
    return future.wait() if wait else future
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/service.py", line 208, in callback
    self._check_fault(response)
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
    body = self._internal_read_body(response)
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
    self._connection.check_xml_content_type(response)
  File "/usr/lib64/python3.8/site-packages/ovirtsdk4/__init__.py", line 846, in check_xml_content_type
    return self._check_content_type(
  File "/usr/lib64/python3.8/site-
[error.log](https://github.com/oVirt/ovirt-engine/files/9135417/error.log)
packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
    raise Error(msg)
ovirtsdk4.Error: The response content type 'text/html;charset=UTF-8' isn't the expected XML
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "fetch_nested": false,
            "follow": [
                "gluster_volumes"
            ],
            "nested_attributes": [],
            "pattern": "name=ocpcluster"
        }
    },
    "msg": "The response content type 'text/html;charset=UTF-8' isn't the expected XML"
}

and the ovirt-engine.log shows large stackdumps with this as the initial cause:

2022-07-18 15:39:34,009-04 ERROR [io.undertow.request] (default task-7) UT005023: Exception handling request to /ovirt-engine/api/v4/clusters: java.lang.RuntimeException: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Problem following 'gluster_volumes' link in Clusters entity.

I've uploaded the full stackdump log. I can call the the API directly:

$ curl -v -H "Authorization: Bearer 2tzYQYrVTMfTqlu0cNVY8OpjGpj9O6...........zUg6XUw" https://manager.example.org/ovirt-engine/api/clusters/ccc53763-c479-410f-af0b-ec846929b46f?follow=gluster_volumes

And this does respond ok.

[bit4man]

error.log

[bennyz]

looks like https://bugzilla.redhat.com/show_bug.cgi?id=2079361

[bit4man]

looks like https://bugzilla.redhat.com/show_bug.cgi?id=2079361

Sure does. Thanks!

[bit4man]

looks like https://bugzilla.redhat.com/show_bug.cgi?id=2079361

One small difference. I can query using follow=gluster_volumes just fine. This BZ seems to hint that this parameter is the problem; I don't use gluster-ovirt nodes, so it doesn't really matter - however it doesn't fail for me, when calling the API directly. Only through Ansible.

[bennyz]

maybe @oliel can clarify if it's the same issue

[michalskrivanek]

looks like https://bugzilla.redhat.com/show_bug.cgi?id=2079361

One small difference. I can query using follow=gluster_volumes just fine. This BZ seems to hint that this parameter is the problem; I don't use gluster-ovirt nodes, so it doesn't really matter - however it doesn't fail for me, when calling the API directly. Only through Ansible.

per https://bugzilla.redhat.com/show_bug.cgi?id=2079361#c1 it is happening on Virt-only clusters.

[bit4man]

per https://bugzilla.redhat.com/show_bug.cgi?id=2079361#c1 it is happening on Virt-only clusters.

I don't think so - they look different. https://bugzilla.redhat.com/show_bug.cgi?id=2079361#c1 talks about "not returned". That's incorrect. When I "curl" manually I do get a gluster_volume returned - just empty. Ie. this succeeds:

$ curl -H "Authorization: Bearer YIE2D......KYCDQ" https://ovirt.example.com/ovirt-engine/api/clusters/ccc53763-c479-410f-af0b-ec846929b46f?follow=gluster_volumes | grep gluster
<link href="/ovirt-engine/api/clusters/ccc53763-c479-410f-af0b-ec846929b46f/refreshglusterhealstatus" rel="refreshglusterhealstatus"/>
    <link href="/ovirt-engine/api/clusters/ccc53763-c479-410f-af0b-ec846929b46f/glusterhooks" rel="glusterhooks"/>
   <link href="/ovirt-engine/api/clusters/ccc53763-c479-410f-af0b-ec846929b46f/glustervolumes" rel="glustervolumes"/>
       <skip_if_gluster_bricks_up>false</skip_if_gluster_bricks_up>
        <skip_if_gluster_quorum_not_met>false</skip_if_gluster_quorum_not_met>
    <gluster_service>false</gluster_service>
   <gluster_volumes/>

So it doesn't fit the idea that "gluster_volumes doesn't work". It seems the call done from the ansible collection/python is not done according to required specs, but I'm having a hard time figuring out what the call actually looks like.

EDIT: Bad paste