DMTF / Redfish-Usecase-Checkers

This is a collection of python3 tools to exercise and validate common use cases for Redfish
Other
1 stars 6 forks source link

Execution of the one_time_boot using profile-simulator throws an error: #24

Closed maximyep closed 5 years ago

maximyep commented 5 years ago

Test execution fails with error (below).

max@myepanes-bmc:~/openbmc/dmtf/Redfish-Usecase-Checkers/one_time_boot$ python3 one_time_boot.py 127.0.0.1:5000 Once Pxe -u root -p password123456
Traceback (most recent call last):
  File "one_time_boot.py", line 225, in <module>
    sys.exit(main(sys.argv))
  File "one_time_boot.py", line 197, in main
    base_url=args.rhost, username=args.user, password=args.password, default_prefix="/redfish/v1")
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 1027, in redfish_client
    max_retry=max_retry)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 952, in __init__
    max_retry=max_retry)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 422, in __init__
    self.get_root_object()
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 553, in get_root_object
    raise excp
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 551, in get_root_object
    resp = self.get('%s%s' % (self.__url.path, self.default_prefix))
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 585, in get
    headers=headers)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 979, in _rest_request
    args=args, body=body, headers=headers)
  File "/home/max/.local/lib/python3.6/site-packages/redfish/rest/v1.py", line 844, in _rest_request
    raise RetriesExhaustedError()
redfish.rest.v1.RetriesExhaustedError

The same error is observed for the OpenBmc QEMU: python3 one_time_boot.py 192.168.122.52:443 Once Pxe -u root -p 0penBmc

tomasg2012 commented 5 years ago

Seems that it's not able to contact whatever servers you're throwing at it, and this is what the "redfish" library is spitting out as a result. The program should be programmed to catch these errors, but are you able to contact these servers on their own?

edit: Sorry I just noticed, the schema "http://" or "https://" is required in the url. This is a requirement of the program and its use of the "redfish" library and I must've overlooked it, it should not require it, or at least not change the default behavior.

maximyep commented 5 years ago

Thanks for the idea (to use https://)! The test starts to execution if the following line was used: one_time_boot_check.py https:// 192.168.XXX.XXX:443 Disabled|Once None|Pxe -u ** -p *** 2019-03-29 11_33_34-2019-03-29 11_24_59-myepanes-bmc  Running  - Oracle VM VirtualBox png - Paint

Test do changes in /redfish/v1/Systems/Self from "Disabled None" to "Once Pxe" but fails with:

No system specified, defaulting to single system, ['Self']
Boot change patch success
Boot status change Disabled None
####sutUri :  /redfish/v1/Systems/Self
####decoded:  {'error': {'@Message.ExtendedInfo': [{'@odata.type': '#Message.v1_0_4.Message', 'Message': 'While attempting to establish a connection to /redfish/v1/Systems/Self, the service was denied access.', 'MessageArgs': ['/redfish/v1/Systems/Self'], 'MessageId': 'Security.1.0.AccessDenied', 'Resolution': 'Attempt to ensure that the URI is correct and that the service has the appropriate credentials.', 'Severity': 'Critical'}], 'code': 'Security.1.0.AccessDenied', 'message': 'While attempting to establish a connection to /redfish/v1/Systems/Self, the service was denied access.'}}
Traceback (most recent call last):
  File "/home/max/redfish/Redfish-Usecase-Checkers/one_time_boot/one_time_boot_check.py", line 72, in <module>
    sys.exit(main(sys.argv))
  File "/home/max/redfish/Redfish-Usecase-Checkers/one_time_boot/one_time_boot_check.py", line 39, in main
    args.target_systems, redfish_obj, args.override_enable, args.typeboot_target, args.delay)
  File "/home/max/redfish/Redfish-Usecase-Checkers/one_time_boot/one_time_boot.py", line 57, in perform_one_time_boot
    *handleBootToSystem(context, systems_url, system, override_enable, typeboot_target, delay),
  File "/home/max/redfish/Redfish-Usecase-Checkers/one_time_boot/one_time_boot.py", line 121, in handleBootToSystem
    newOverride, newType = decoded['Boot']['BootSourceOverrideEnabled'], decoded['Boot']['BootSourceOverrideTarget']
KeyError: 'Boot'
maximyep commented 5 years ago

Closed by mistake

billdodd commented 5 years ago

https://github.com/DMTF/Redfish-Usecase-Checkers/blob/9e51c25cec87f8d9cda0a0c54ed02e017c04ac4a/one_time_boot/one_time_boot.py#L109-L120

I'm pretty sure the error occured in the above code when the response from a GET request returned a 4XX or 5XX status and an extended error in the payload as seen in the output above:

{
    'error': {
        '@Message.ExtendedInfo': [
            'Message': 'While attempting to establish a connection to /redfish/v1/Systems/Self, the service was denied access.'
            ...
        ]
    }
}

And then the tool tried to read the BootSourceOverride properties from that payload and got the KeyError: 'Boot' error.

The code should be hardened to only try to read the properties from the response if the response status is 200.

billdodd commented 5 years ago

I looked at this again in more detail. I think the root cause is that the tool is using session authentication and then performing a reboot. After the system reboots the previous session is no longer valid. But the tool is still using that session to check the state of the 'Boot' properties after reboot.

I think the tool probably needs to be changed to remove the --auth arg and always use basic auth for this use case.

tomasg2012 commented 5 years ago

While the tool should be hardened, isn't that a simulator problem? No reason for the Session to be invalidated on a system reboot, which is independent of Redfish and its controller.

billdodd commented 5 years ago

My bad. I should have looked even more closely. Agreed that the system reboot should not cause the session to be invalidated.

But I don't think this is a profile-simulator issue. The session in the profile-simulator does not get invalidated or timeout as best as I can tell. I don't even think the profile simulator was being used when the KeyError: 'Boot' error occurred. Note that when the issue was originally opened the host:port was 127.0.0.1:5000 (profile-simulator). But later, when the KeyError: 'Boot' error was reported the host:port was 192.168.XXX.XXX:443. Further, the profile-simulator (at least in its default config) doesn't emit the @Message.ExtendedInfo seen in the Mar 29 error output.

So, what is the real cause? Perhaps the session timed out? It looks like we sleep for no longer than 30 seconds in the loop. If the session timeout is set to 30 seconds, it may be just a bit longer than 30 seconds of inactivity on the session with the current code logic.