search

zhmcclient / python-zhmcclient

A pure Python client library for the IBM Z HMC Web Services API
Apache License 2.0
38 stars 26 forks source link

Unable to find storage volume by UUID #712

Closed ltrager closed 3 years ago

ltrager commented 3 years ago

Describe the bug I am working to enable IBM Z in MAAS, which is a tool to automatically configure and deploy operating systems. This is done by network booting into an ephemeral environment and writing the operating system to disk. IBM Z does not have a boot order so MAAS must use python-zhmcclient to reconfigure which device the partition should be booted from.

Configuring a partition to boot over the network is easy

nic = partition.nics.find(**{"mac-address": mac_address})
partition.update_properties({
    boot-device': 'network-adapter',
    'boot-network-device': nic.uri,
})

To Reproduce Configuring a system to boot from a storage volume is where I run into trouble. MAAS knows the UUID of the storage volume but trying to find it it results in a 500 error

for storage_group in partition.list_attached_storage_groups():
    try:
        storage_volume = storage_group.storage_volumes.find(uuid=uuid)
    except NotFound:
        pass
    else:
        partition.update_properties({
            'boot-device': 'storage-volume',
            'boot-storage-volume': storage_volume.uri,
        })
        break
eyError                                  Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in get_property(self, name)
    242         try:
--> 243             return self._properties[name]
    244         except KeyError:

KeyError: 'name'

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
<ipython-input-11-022b961f15a3> in <module>
----> 1 storage_group.storage_volumes.find(uuid="6005076306ffd39f0000000000000011")

<decorator-gen-118> in find(self, **filter_args)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_manager.py in find(self, **filter_args)
    757         num_objs = len(obj_list)
    758         if num_objs == 0:
--> 759             raise NotFound(filter_args, self)
    760         if num_objs > 1:
    761             raise NoUniqueMatch(filter_args, self, obj_list)

~/.local/lib/python3.8/site-packages/zhmcclient/_exceptions.py in __init__(self, filter_args, manager)
   1093         if parent:
   1094             in_str = " in {} {!r}". \
-> 1095                 format(parent.__class__.__name__, parent.name)
   1096         else:
   1097             in_str = ""

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in name(self)
    155         # We avoid storing the name in an instance variable, because it can
    156         # be modified via update_properties().
--> 157         return self.get_property(self.manager._name_prop)
    158 
    159     @property

<decorator-gen-122> in get_property(self, name)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in get_property(self, name)
    245             if self._full_properties:
    246                 raise
--> 247             self.pull_full_properties()
    248             return self._properties[name]
    249 

<decorator-gen-121> in pull_full_properties(self)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in pull_full_properties(self)
    203           :exc:`~zhmcclient.ConnectionError`
    204         """
--> 205         full_properties = self.manager.session.get(self._uri)
    206         self._properties = dict(full_properties)
    207         self._properties_timestamp = int(time.time())

<decorator-gen-134> in get(self, uri, logon_required)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_session.py in get(self, uri, logon_required)
    790                                   format(msg), HTTPError(result_object))
    791         result_object = _result_object(result)
--> 792         raise HTTPError(result_object)
    793 
    794     @logged_api_call

HTTPError: 500,12: Unexpected Throwable [GET /api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1]

Expected behavior storage_group.storage_volumes.find(uuid=uuid) should return the storage volume.

Ideal behavior Ideally you wouldn't have to iterate over partition.list_attached_storage_groups() you could do something like partition.storage_volumes.find(uuid=uuid)

Environment information

andy-maier commented 3 years ago

Hi Lee. I'll look into this.

andy-maier commented 3 years ago

@ltrager Lee, I could not reproduce this behavior on a z14-LR1 I have access to, neither with an existing UUID nor with a non-existing UUID.

An HTTP status 500 always means the server side has an issue. In case of 500,12 it is clearly an internal error in the HMC code.

Can you please post here which machine model this happens on, and which bundle version you have on the HMC, then I can probably quickly find out whether there is already a newer bundle with a fix for that. I recall encountering that error symptom before, but it is pretty generic so it may or may not be the same error.

If there is no newer bundle with a fix, you would need to open a PMR. I can notify the colleagues in the HMC dev team in parallel once you tell me the PMR number.

frank-heimes commented 3 years ago

Hi Andy, let me jump in on that. It's a z14 GA2 (located in the Dallas System Center) we have remote access to. Machine type - model: | 3906 - M04 Support element version: | 2.14.1 (P41414.300) System mode: | Dynamic Partition Manager

Taken from System Information: EC number: ?????? LIC control level: 0005 Engineering Changes AROM Type: 3906 Model number: M04 Serial number: ?????????? Version: 2.14.1 Driver level: 36 Bundle level: S50

If you need more details from the "Internal Code Change Information" and/or the above info that I replaced by "?", let me know.

Thx, Frank

andy-maier commented 3 years ago

Hi Frank, Is this z14 system managed by a z14 HMC or by a z15 HMC? In case of a z14 HMC, the latest bundle would be T28.

I suggest that you ask the people managing the machine to upgrade at least the HMC and more ideally everything to the latest available bundle level, and then see whether that fixes the problem.

To Lee: In parallel, we can try to investigate it. For that, please post the entire request to and response from the HMC that resulted in the 500,12 error. That can be obtained by turning on the Python logger named "zhmcclient.hmc" to write the logs to stdout, e.g. like so:

import logging

handler = logging.StreamHandler()
format_string = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handler.setFormatter(logging.Formatter(format_string))
logger = logging.getLogger('zhmcclient.hmc')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
vmorris commented 3 years ago

Hi @andy-maier .. this is a z14 HMC, and the installed bundle is H37. We plan on upgrading it to H39 today.

Just a point of clarification.. isn't T# bundles for TKE?

vmorris commented 3 years ago

I was tinkering with this API (and importantly on this same HMC) myself recently and came up with the following snippet to set the boot-storage-volume to the first boot disk found in the first storage group... it's not really a great approach, but it worked for what I was wanting to do:

    # change the partition properties to boot from the first boot volume found in the first storage group
    try:
        storage_group = partition.list_attached_storage_groups()[0]
    except IndexError:
        print(f"Could not detect any storage groups attached to partition {partition}")
        exit(1)

    storage_volume_manager = storage_group.storage_volumes
    volumes = storage_volume_manager.list()

    boot_volume = None
    for volume in volumes:
        if volume.get_property("usage") == "boot":
            boot_volume = volume
            break

    if not boot_volume:
        print(f"Could not detect a boot volume")
        exit(1)

    boot_uri = boot_volume.get_property("element-uri")

    partition.update_properties(
        {
            "boot-device": "storage-volume",
            "boot-storage-volume": boot_uri,
        }
    )
    print(
        f"Updated partition {partition_name} on {cpc_name} to boot storage volume {boot_uri}."
    )
ltrager commented 3 years ago

@vmorris - thanks for updating the firmware. I still get the same error as before.

@andy-maier - Below is the debug output

In [8]: storage_group.storage_volumes.find(uuid="6005076306ffd39f0000000000000011")                    
2021-03-04 20:26:49,653 - zhmcclient.hmc - DEBUG - Request: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes, headers: {'Content-type': 'application/json', 'Accept': '*/*', 'X-API-Session': '********'}, content(max.1000): None
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.29.120.16'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2021-03-04 20:26:49,710 - zhmcclient.hmc - DEBUG - Respons: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes, status: 200, headers: {'Strict-Transport-Security': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "img-src * data:; frame-ancestors 'self'", 'x-request-id': 'Sxd06cc08a-7d17-11eb-aaf6-00106f0ddae9.3 Rx4', 'Cache-control': 'no-cache', 'Content-Type': 'application/json;charset=UTF-8', 'Content-Length': '441', 'Date': 'Fri, 05 Mar 2021 02:31:16 GMT', 'Keep-Alive': 'timeout=20', 'Connection': 'keep-alive', 'Server': 'Hardware management console API web server / 2.0'}, content(max.1000): b'{"storage-volumes":[{"element-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab32a2-6051-11eb-a7af-00106f23e4b1","fulfillment-state":"complete","size":40.0,"usage":"data","name":"40.00 GiB Data"},{"element-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab348c-6051-11eb-a7af-00106f23e4b1","fulfillment-state":"complete","size":40.0,"usage":"boot","name":"40.00 GiB Boot"}]}'
2021-03-04 20:26:49,710 - zhmcclient.hmc - DEBUG - Request: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab32a2-6051-11eb-a7af-00106f23e4b1, headers: {'Content-type': 'application/json', 'Accept': '*/*', 'X-API-Session': '********'}, content(max.1000): None
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.29.120.16'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2021-03-04 20:26:49,839 - zhmcclient.hmc - DEBUG - Respons: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab32a2-6051-11eb-a7af-00106f23e4b1, status: 200, headers: {'Strict-Transport-Security': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "img-src * data:; frame-ancestors 'self'", 'x-request-id': 'Sxd06cc08a-7d17-11eb-aaf6-00106f0ddae9.3 Rx5', 'Cache-control': 'no-cache', 'Content-Type': 'application/json;charset=UTF-8', 'Content-Length': '1930', 'Date': 'Fri, 05 Mar 2021 02:31:16 GMT', 'Keep-Alive': 'timeout=20', 'Connection': 'keep-alive', 'Server': 'Hardware management console API web server / 2.0'}, content(max.1000): b'{"parent":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1","element-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab32a2-6051-11eb-a7af-00106f23e4b1","fulfillment-state":"complete","size":40.0,"paths":[{"target-world-wide-port-name":"500507630608d39f","logical-unit-number":"4000401200000000","device-number":"0009","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630618939f","logical-unit-number":"4000401200000000","device-number":"0008","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630613d39f","logical-unit-number":"4000401200000000","device-number":"0009","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630603939f","logical-unit-number":"4000401200000000","device-number":"0008","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"targe
2021-03-04 20:26:49,839 - zhmcclient.hmc - DEBUG - Request: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab348c-6051-11eb-a7af-00106f23e4b1, headers: {'Content-type': 'application/json', 'Accept': '*/*', 'X-API-Session': '********'}, content(max.1000): None
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.29.120.16'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2021-03-04 20:26:50,004 - zhmcclient.hmc - DEBUG - Respons: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab348c-6051-11eb-a7af-00106f23e4b1, status: 200, headers: {'Strict-Transport-Security': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "img-src * data:; frame-ancestors 'self'", 'x-request-id': 'Sxd06cc08a-7d17-11eb-aaf6-00106f0ddae9.3 Rx6', 'Cache-control': 'no-cache', 'Content-Type': 'application/json;charset=UTF-8', 'Content-Length': '1930', 'Date': 'Fri, 05 Mar 2021 02:31:16 GMT', 'Keep-Alive': 'timeout=20', 'Connection': 'keep-alive', 'Server': 'Hardware management console API web server / 2.0'}, content(max.1000): b'{"parent":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1","element-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1/storage-volumes/4fab348c-6051-11eb-a7af-00106f23e4b1","fulfillment-state":"complete","size":40.0,"paths":[{"target-world-wide-port-name":"500507630608d39f","logical-unit-number":"4000401100000000","device-number":"0009","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630618939f","logical-unit-number":"4000401100000000","device-number":"0008","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630613d39f","logical-unit-number":"4000401100000000","device-number":"0009","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"target-world-wide-port-name":"500507630603939f","logical-unit-number":"4000401100000000","device-number":"0008","partition-uri":"/api/partitions/beda4c72-7a50-11ea-b239-00106f23e4b1"},{"targe
2021-03-04 20:26:50,004 - zhmcclient.hmc - DEBUG - Request: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1, headers: {'Content-type': 'application/json', 'Accept': '*/*', 'X-API-Session': '********'}, content(max.1000): None
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.29.120.16'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2021-03-04 20:26:50,081 - zhmcclient.hmc - DEBUG - Respons: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1, status: 500, headers: {'Strict-Transport-Security': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "img-src * data:; frame-ancestors 'self'", 'x-request-id': 'Sxd06cc08a-7d17-11eb-aaf6-00106f0ddae9.3 Rx7', 'Cache-control': 'no-cache', 'x-api-reason-code': '12', 'Content-Type': 'application/json', 'Content-Length': '4389', 'Date': 'Fri, 05 Mar 2021 02:31:16 GMT', 'Connection': 'close', 'Server': 'Hardware management console API web server / 2.0'}, content(max.1000): b'{"http-status":500,"reason":12,"request-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1","request-method":"GET","message":"Unexpected Throwable","stack":"ApiExceptionInternalServerError: Unexpected Throwable\\n\\tstack trace suppressed\\njava.lang.NullPointerException\\n\\tat com.ibm.hwmca.z.dam.api.StorageGroupProvider.getProperties(StorageGroupProvider.java:1704)\\n\\tat com.ibm.hwmca.z.dam.api.DamApiProvider.getProperties(DamApiProvider.java:986)\\n\\tat com.ibm.hwmca.z.dam.api.DamApiProvider.getProperties(DamApiProvider.java:937)\\n\\tat com.ibm.hwmca.z.dam.api.StorageGroupProvider.service(StorageGroupProvider.java:405)\\n\\tat com.ibm.hwmca.fw.api.RequestDispatcher.service(RequestDispatcher.java:1009)\\n\\tat com.ibm.hwmca.fw.api.RequestDispatcher.service(RequestDispatcher.java:824)\\n\\tat com.ibm.hwmca.fw.api.ApiServlet.execute(ApiServlet.java:129)\\n\\tat com.ibm.hwmca.fw.api.ApiServlet.doGet(ApiServlet.java:108)\\n\\tat javax.servlet.http.HttpServlet.service(H
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in get_property(self, name)
    242         try:
--> 243             return self._properties[name]
    244         except KeyError:

KeyError: 'name'

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
<ipython-input-8-022b961f15a3> in <module>
----> 1 storage_group.storage_volumes.find(uuid="6005076306ffd39f0000000000000011")

<decorator-gen-118> in find(self, **filter_args)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_manager.py in find(self, **filter_args)
    757         num_objs = len(obj_list)
    758         if num_objs == 0:
--> 759             raise NotFound(filter_args, self)
    760         if num_objs > 1:
    761             raise NoUniqueMatch(filter_args, self, obj_list)

~/.local/lib/python3.8/site-packages/zhmcclient/_exceptions.py in __init__(self, filter_args, manager)
   1093         if parent:
   1094             in_str = " in {} {!r}". \
-> 1095                 format(parent.__class__.__name__, parent.name)
   1096         else:
   1097             in_str = ""

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in name(self)
    155         # We avoid storing the name in an instance variable, because it can
    156         # be modified via update_properties().
--> 157         return self.get_property(self.manager._name_prop)
    158 
    159     @property

<decorator-gen-122> in get_property(self, name)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in get_property(self, name)
    245             if self._full_properties:
    246                 raise
--> 247             self.pull_full_properties()
    248             return self._properties[name]
    249 

<decorator-gen-121> in pull_full_properties(self)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_resource.py in pull_full_properties(self)
    203           :exc:`~zhmcclient.ConnectionError`
    204         """
--> 205         full_properties = self.manager.session.get(self._uri)
    206         self._properties = dict(full_properties)
    207         self._properties_timestamp = int(time.time())

<decorator-gen-134> in get(self, uri, logon_required)

~/.local/lib/python3.8/site-packages/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/.local/lib/python3.8/site-packages/zhmcclient/_session.py in get(self, uri, logon_required)
    790                                   format(msg), HTTPError(result_object))
    791         result_object = _result_object(result)
--> 792         raise HTTPError(result_object)
    793 
    794     @logged_api_call

HTTPError: 500,12: Unexpected Throwable [GET /api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1]
andy-maier commented 3 years ago

Hi Vance :-) I got the bundle number in a quick talk with a colleague from zFW. Maybe we miscommunicated.

andy-maier commented 3 years ago

@ltrager @frank-heimes The log looks good to me for opening a PMR. The important part is the last request/reponse pair. Even though the truncation to 1000 chars cuts off some of the Java stack trace, it should be sufficient for the dev team. I talked with the DPM architect about the situation and that's what he asked for.

So please go ahead and open a PMR, and copy the entire log output you posted above, and please do mention the HMC bundle version. Please post the PMR number here so I can make sure it gets attention (I'm sure it would get attention nevertheless, but just to double check).

frank-heimes commented 3 years ago

Hi Andy, opening a PMR will be a bit challenging, because when I login to the IBM services pages to do that, I can only open PMRs for systems that I/we own - and we have only a z13 system (even in PR/SM mode). Not sure if I can refer in my PMR to a different system owned by someone else - in this case even IBM itself (and if that's accepted). Anyway, I'll try -- but if not possible or accepted by service maybe Vance and team need to help out and open one...

frank-heimes commented 3 years ago

I've managed to open a PMR: FNTFTDS As description I've attached a PDF from the github issue that incl. all the logs that were shared so far.) As contact (for the machine) I've mentioned Vance. I hope that this will work ...

vmorris commented 3 years ago

I'm not able to duplicate this error myself, in fact I am able to locate the volume by uuid..

uuid = "6005076306FFD39F0000000000000108"
for storage_group in partition.list_attached_storage_groups():
    print(f"Searching for volume with UUID {uuid}")
    try:
        storage_volume = storage_group.storage_volumes.find(uuid=uuid)
    except NotFound:
        pass
    else:
        print(f"Found storage volume with UUID {uuid}.")
        print(storage_volume.uri)

outputs:

Searching for volume with UUID 6005076306FFD39F0000000000000108
Found storage volume with UUID 6005076306FFD39F0000000000000108.
/api/storage-groups/0726bfd4-2ab8-11eb-82e6-00106f23e4b1/storage-volumes/078a260a-2ab8-11eb-82e6-00106f23e4b1

... I wonder if this is some limitation with the access that your ID has :thinking:

I'm on version 

$ python -m zhmcclient.debuginfo
Operating system: Linux 5.10.15-200.fc33.x86_64 on x86_64
Python implementation: CPython 3.9.1 (64 bit, wide unicode)
Python version: 3.9.1
zhmcclient version: 0.29.0.dev1

This is commit ID 4b7cbc462e49574c8a4dedb6ecd83de701891466

ltrager commented 3 years ago

I was using 0.28 from pip however I was able to reproduce it using using ae4c0e514398f6ae97da04c406d86b865acf578d, which is the tip of master.

Another thing I noticed is that I appear to be getting a 500 error even if I do something simple such as storage_group.name

In [8]: storage_group.name                                                                                                                                                                                      
2021-03-05 17:20:09,607 - zhmcclient.hmc - DEBUG - Request: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1, headers: {'Content-type': 'application/json', 'Accept': '*/*', 'X-API-Session': '********'}, content(max.1000): None
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.29.120.16'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2021-03-05 17:20:09,728 - zhmcclient.hmc - DEBUG - Respons: GET https://172.29.120.16:6794/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1, status: 500, headers: {'Strict-Transport-Security': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "img-src * data:; frame-ancestors 'self'", 'x-request-id': 'Sxd06cc08a-7d17-11eb-aaf6-00106f0ddae9.10 Rx4', 'Cache-control': 'no-cache', 'x-api-reason-code': '12', 'Content-Type': 'application/json', 'Content-Length': '4389', 'Date': 'Fri, 05 Mar 2021 23:24:36 GMT', 'Connection': 'close', 'Server': 'Hardware management console API web server / 2.0'}, content(max.1000): b'{"http-status":500,"reason":12,"request-uri":"/api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1","request-method":"GET","message":"Unexpected Throwable","stack":"ApiExceptionInternalServerError: Unexpected Throwable\\n\\tstack trace suppressed\\njava.lang.NullPointerException\\n\\tat com.ibm.hwmca.z.dam.api.StorageGroupProvider.getProperties(StorageGroupProvider.java:1704)\\n\\tat com.ibm.hwmca.z.dam.api.DamApiProvider.getProperties(DamApiProvider.java:986)\\n\\tat com.ibm.hwmca.z.dam.api.DamApiProvider.getProperties(DamApiProvider.java:937)\\n\\tat com.ibm.hwmca.z.dam.api.StorageGroupProvider.service(StorageGroupProvider.java:405)\\n\\tat com.ibm.hwmca.fw.api.RequestDispatcher.service(RequestDispatcher.java:1009)\\n\\tat com.ibm.hwmca.fw.api.RequestDispatcher.service(RequestDispatcher.java:824)\\n\\tat com.ibm.hwmca.fw.api.ApiServlet.execute(ApiServlet.java:129)\\n\\tat com.ibm.hwmca.fw.api.ApiServlet.doGet(ApiServlet.java:108)\\n\\tat javax.servlet.http.HttpServlet.service(H
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/python-zhmcclient/zhmcclient/_resource.py in get_property(self, name)
    242         try:
--> 243             return self._properties[name]
    244         except KeyError:

KeyError: 'name'

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
<ipython-input-8-1663340b685d> in <module>
----> 1 storage_group.name

~/python-zhmcclient/zhmcclient/_resource.py in name(self)
    155         # We avoid storing the name in an instance variable, because it can
    156         # be modified via update_properties().
--> 157         return self.get_property(self.manager._name_prop)
    158 
    159     @property

<decorator-gen-122> in get_property(self, name)

~/python-zhmcclient/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/python-zhmcclient/zhmcclient/_resource.py in get_property(self, name)
    245             if self._full_properties:
    246                 raise
--> 247             self.pull_full_properties()
    248             return self._properties[name]
    249 

<decorator-gen-121> in pull_full_properties(self)

~/python-zhmcclient/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/python-zhmcclient/zhmcclient/_resource.py in pull_full_properties(self)
    203           :exc:`~zhmcclient.ConnectionError`
    204         """
--> 205         full_properties = self.manager.session.get(self._uri)
    206         self._properties = dict(full_properties)
    207         self._properties_timestamp = int(time.time())

<decorator-gen-134> in get(self, uri, logon_required)

~/python-zhmcclient/zhmcclient/_logging.py in log_api_call(func, *args, **kwargs)
    221                                 log_escaped(repr(kwargs))))
    222 
--> 223         result = func(*args, **kwargs)
    224 
    225         if _log_it:

~/python-zhmcclient/zhmcclient/_session.py in get(self, uri, logon_required)
    790                                   format(msg), HTTPError(result_object))
    791         result_object = _result_object(result)
--> 792         raise HTTPError(result_object)
    793 
    794     @logged_api_call

HTTPError: 500,12: Unexpected Throwable [GET /api/storage-groups/4f59da24-6051-11eb-a7af-00106f23e4b1]
andy-maier commented 3 years ago

An HTTP status 500 is not related to missing object access and instead always indicates an issue on the HMC, i.e. it is always worth opening a defect for.

In case of missing object or task permissions, the typical HTTP status is 403 with reason 1, or also 404 in cases where it is documented that the resource without access is simply not part of the result (but then that user should not get a URI for the object in the first place, however the user may have obtained the URI elsewhere). Anyway, if 500 comes back -> defect.

Having said that, I have no explanation as to why Vance cannot reproduce the HTTP status 500, but being a defect it may have all kinds of reasons and the different behavior may indeed be caused by using different userids. Nevertheless: 500 -> defect.

andy-maier commented 3 years ago

@frank-heimes Frank, the PMR IDs tend to be numbers, separated by comma. Can you please double check whether you find such a comma separated number sequence in the PMR?

The HMC dev team has recently received a similar error (at least it had the same symptom) on a z14 GA2. A fix is being worked, but the target bundle is not yet clear and also it is not confirmed that it is really the same error. So your PMR needs to be looked at in any case, to be sure.

andy-maier commented 3 years ago

@ltrager Could you please check whether your userid has object access to those FICON adapters on the CPC that are configured as candidate adapters for the storage group that fails? If not, then please establish that access and see if that fixes the issue.

@vmorris Vance, this may be the auth problem you suspected :-)

frank-heimes commented 3 years ago

Hi @andy-maier - well, that is the 'code' I got and it's still mentioned in the IBM Service Report: IBM problem number: FNTFTDS I'll send you a 'print to pdf' of the entire service request by mail ...

frank-heimes commented 3 years ago

@andy-maier Just yesterday I had a look at my user definition in that system (which should be equal to Lee's) and grabbed the data for documentation purposes (because at some point we also need to document how a HMC user ids needs to be setup that allow MAAS DPM/LPAR usage via zhmcclient) and I grabbed this (I stroke out some details with '???'): " Authentication Authentication type: Local Password rule: ??????? Multi-factor authentication enabled: No

Roles usergroup

Groups

Tasks Access Removable Media Activate Adapter Details Configure Channel Path On/Off Configure Storage - System Programmer Customize Scheduled Operations Customize/Delete Activation Profiles Deactivate Dump Export WWPNs Image Details Initiate zVM Management Guest Dump Integrated 3270 Console Integrated ASCII Console Load Load from Removable Media or Server Logical Processor Add Manage Storage Resources Manage Web Services API Logs Monitor System Events Operating System Messages Partition Details Partition Details - Controls PSW Restart Reassign Devices Report a Problem Reset Clear Reset Normal Single Object Operations Start Start All Processors Stop Stop All Processors Storage Resource Details View Activation Profiles View Console Events View Console Information View Console Service History

Object Types

Objects P1234567 (Defined CPC) P1234567:????????? (Storage Template) P1234567:????????? MAAS Storage Group (FCP Storage Group) P1234567:HYPERS1 (HiperSockets Adapter) P1234567:??04-????????? (FCP Storage Group) P1234567:??04-????????? (Partition) P1234567:??04???? (LPAR Image) P1234567:??05-????????? (FCP Storage Group) P1234567:??05-????????? (Partition) P1234567:??05???? (LPAR Image) P1234567:??06-????????? (FCP Storage Group) P1234567:??06-????????? (Partition) P1234567:??06???? (LPAR Image) " I can't say if something comes on top that is 'derived' by the "usergroup", since I don't have permissions to see the details and the definition of that group.

In other words: NO access to Object type "FICON Adapter" - at least not visible (nothing listed under object type in our account definition).

@vmorris Since we (Lee and me) cannot modify our own accounts I would need to ask you to modify them by grating Object Type access to "FICON Adapter", as suggested by @andy-maier

vmorris commented 3 years ago

I added the FICON adapters to the list of objects that you role has access to.

ltrager commented 3 years ago

@vmorris that worked! I think the HMC should be returning 403 in this case with a message about what permission is missing.

Thanks for your help!

frank-heimes commented 3 years ago

Glad to read this ! Thx @vmorris for the modification (adding FICON ObjectType) and thx @andy-maier for the hint !

andy-maier commented 3 years ago

I finally understood the FNTFTDS number. IBM introduced a new system for handling client problems, and that is the client facing ID. Internally, there is still a PMR created in RETAIN. For the FNTFTDS problem, the PMR (HW) number is 22883,004,000, just for the record, but you don't need that number.

frank-heimes commented 3 years ago

Hi @andy-maier - okay that explains it. Thx for shedding some light into it (I also wondered a bit when I saw that code instead of number ...)

andy-maier commented 3 years ago

I am closing this PR as "pursued elsewhere".

If you have further occurrences of this problem, feel free to reopen this issue, or to open a new one, whichever you prefer.

frank-heimes commented 3 years ago

@andy-maier Many thx for your support on this!