search

openbmc / openbmc-test-automation

Apache License 2.0
100 stars 92 forks source link

empty report by redfish/dmtf_tools/Redfish_Usecase_Checkers.robot 'Test BMC Redfish Power Control Usecase' #2177

Closed generatz closed 2 years ago

generatz commented 2 years ago

We just experienced a hardware issue with one of the systems we're testing such that the DMTF Redfish Usecase test fails as follows:

$ python3 Redfish-Usecase-Checkers/power_control/power_control.py -r xx.xx.xx.xx:443 -u root -p 0penBmcXXX -S Always ; echo "rc=$?"
Found 1 system instances
Resetting system using On
Monitoring power state for system...
ERROR: system was not in the On state after using On as the reset type.
Resetting system using ForceOff
Monitoring power state for system...
Resetting system using ForceOn
Monitoring power state for system...
ERROR: system was not in the On state after using ForceOn as the reset type.
Resetting system using ForceRestart
Monitoring power state for system...
ERROR: system was not in the On state after using ForceRestart as the reset type.
Resetting system using GracefulRestart
Monitoring power state for system...
ERROR: system was not in the On state after using GracefulRestart as the reset type.
Resetting system using GracefulShutdown
Monitoring power state for system...
Resetting system using PowerCycle
Monitoring power state for system...
ERROR: system was not in the On state after using PowerCycle as the reset type.
rc=1

The failed test exposes the following crash in redfish/dmtf_tools/Redfish_Usecase_Checkers.robot, as reported in the log:

Traceback (most recent call last):
  File "/home/ojenkins/workspace/icebridge_redfish_dmtf_tools/lib/gen_cmd.py", line 427, in shell_cmd
    BuiltIn().fail(err_msg)
  File "/home/ojenkins/workspace/icebridge_redfish_dmtf_tools/temp-python/lib/python3.6/site-packages/robot/libraries/BuiltIn.py", line 534, in fail
    raise AssertionError(msg) if msg else AssertionError()

As a consequence, no useful information is reported about the actual problem.

Here's my patch to show how I updated redfish/dmtf_tools/Redfish_Usecase_Checkers.robot to work around the crash and to report the test case results, as follows:

$ git diff redfish/dmtf_tools/Redfish_Usecase_Checkers.robot
diff --git a/redfish/dmtf_tools/Redfish_Usecase_Checkers.robot b/redfish/dmtf_tools/Redfish_Usecase_Checkers.robot
index 49304085..fdfa369e 100644
--- a/redfish/dmtf_tools/Redfish_Usecase_Checkers.robot
+++ b/redfish/dmtf_tools/Redfish_Usecase_Checkers.robot
@@ -23,7 +23,7 @@ ${command_account}        ${DEFAULT_PYTHON} ${rsv_dir_path}${/}account_managemen

 ${command_power_control}  ${DEFAULT_PYTHON} ${rsv_dir_path}${/}power_control/power_control.py
 ...                       -r ${OPENBMC_HOST}:${HTTPS_PORT} -u ${OPENBMC_USERNAME}
-...                       -p ${OPENBMC_PASSWORD} -S Always
+...                       -p ${OPENBMC_PASSWORD} -S Always -d ${EXECDIR}${/}logs${/}

 ${power_on_timeout}       7 mins
 ${power_off_timeout}      7 mins
@@ -83,6 +83,33 @@ Test Teardown Execution
 DMTF Power
     [Documentation]  Power the BMC machine on via DMTF tools.

-    Print Timen  Doing "DMTF Power".
+    Print Timen  Running "DMTF Power".
+    ${output}=  Run DMTF Tool  ${rsv_dir_path}  ${command_power_control}  check_error=1
+    Log  ${output}
+
+    ${json}=  OperatingSystem.Get File    ${EXECDIR}${/}logs${/}results.json
+
+    ${object}=  Evaluate  json.loads('''${json}''')  json
+
+    ${result_list}=  Set Variable  ${object["TestResults"]}
+
+    @{failed_tc_list}=    Create List
+    @{error_messages}=    Create List
+
+    Log   result: ${result_list}
+
+    FOR  ${result}  IN  @{result_list}
+       ${rc}=    evaluate    'ErrorMessages'=='${result}'
+       ${num}=  Run Keyword If  ${rc} == False  Set Variable  ${result_list["${result}"]["fail"]}
+       Run Keyword If  ${num} != None and ${num} > 0  Append To List  ${failed_tc_list}   ${result}
+       ${error_messages}=  Run Keyword If  ${rc} == True   Set Variable 
+       ...   ${result_list["ErrorMessages"]}
+    END
+
+    Log Many                   ErrorMessages:   @{error_messages}
+    Log To Console     ErrorMessages:
+    FOR   ${msg}  IN  @{error_messages}
+       Log To Console  ${msg}
+    END

-    Run DMTF Tool  ${rsv_dir_path}  ${command_power_control}
+    Should Be Empty  ${error_messages}   DMTF Power keyword failed.

Here's the resulting modified Keyword:

DMTF Power
    [Documentation]  Power the BMC machine on via DMTF tools.

    Print Timen  Running "DMTF Power".
    ${output}=  Run DMTF Tool  ${rsv_dir_path}  ${command_power_control}  check_error=1
    Log  ${output}

    ${json}=  OperatingSystem.Get File    ${EXECDIR}${/}logs${/}results.json

    ${object}=  Evaluate  json.loads('''${json}''')  json

    ${result_list}=  Set Variable  ${object["TestResults"]}

    @{failed_tc_list}=    Create List
    @{error_messages}=    Create List

    Log   result: ${result_list}

    FOR  ${result}  IN  @{result_list}
       ${rc}=    evaluate    'ErrorMessages'=='${result}'
       ${num}=  Run Keyword If  ${rc} == False  Set Variable  ${result_list["${result}"]["fail"]}
       Run Keyword If  ${num} != None and ${num} > 0  Append To List  ${failed_tc_list}   ${result}
       ${error_messages}=  Run Keyword If  ${rc} == True   Set Variable
       ...   ${result_list["ErrorMessages"]}
    END

    Log Many            ErrorMessages:   @{error_messages}
    Log To Console      ErrorMessages:
    FOR   ${msg}  IN  @{error_messages}
       Log To Console   ${msg}
    END

    Should Be Empty  ${error_messages}   DMTF Power keyword failed.
gkeishin commented 2 years ago

Looking into it.. Thank you for the suggestion here.

gkeishin commented 2 years ago

https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-test-automation/+/52328