openbmc / openbmc-test-automation

Apache License 2.0
100 stars 92 forks source link

multiple fixes for comparisons and loops in security/test_bmc_connections.robot and minor enhancements #2184

Closed generatz closed 2 years ago

generatz commented 2 years ago

In security/test_bmc_connections.robot the testcase Flood Patch Without Auth Token And Check Stability Of BMC is failing because Should Be Equal is used to compare what it finds are types <class 'int'> and <class 'str'>. I've fixed that by instead using Should Be Equal As Integers.

Secondly, it does not complete the desired Login And Create User after the last set of posts because the loop count is not correctly set. As a result, when using ${iterations} = 100 (as I typically do for a smoke test) it will always pass.

Additionally, I've slightly enhanced the output report to indicate where in the loop the errors occur, to facilitate debugging.

Finally, I also applied the changes above to Flood Post Without Auth Token And Check Stability Of BMC and similarly fixed the iteration value in Test Stability On Large Number Of Wrong Login Attempts To GUI

Patch file:

diff --git a/security/test_bmc_connections.robot b/security/test_bmc_connections.robot
index 07b5364c..d877edd8 100644
--- a/security/test_bmc_connections.robot
+++ b/security/test_bmc_connections.robot
@@ -48,12 +48,12 @@ Test Patch Without Auth Token Fails
 Flood Patch Without Auth Token And Check Stability Of BMC
     [Documentation]  Flood patch method without auth token and check BMC stability.
     [Tags]  Flood_Patch_Without_Auth_Token_And_Check_Stability_Of_BMC
-    @{status_list}=  Create List

+    @{fail_list}=  Create List
     ${active_channel_config}=  Get Active Channel Config
     ${ethernet_interface}=  Set Variable  ${active_channel_config['${CHANNEL_NUMBER}']['name']}

-    FOR  ${i}  IN RANGE  ${1}  ${iterations}
+    FOR  ${i}  IN RANGE  ${1}  ${iterations} + 1
         Log To Console  ${i}th iteration
         Run Keyword And Ignore Error
         ...  Redfish.Patch  ${REDFISH_NW_ETH_IFACE}${ethernet_interface}  body={'HostName': '${hostname}'}
@@ -61,13 +61,14 @@ Flood Patch Without Auth Token And Check Stability Of BMC
         # Every 100th iteration, check BMC allows patch with auth token.
         ${status}=  Run Keyword If  ${i} % 100 == 0  Run Keyword And Return Status
         ...  Login And Configure Hostname
-        Run Keyword If  ${status} == False  Append To List  ${status_list}  ${status}
+        
+        Run Keyword If  ${status} == False  Append To List  ${fail_list}  ${i}
     END
     ${verify_count}=  Evaluate  ${iterations}/100
-    ${fail_count}=  Get Length  ${status_list}
+    ${fail_count}=  Get Length  ${fail_list}

-    Should Be Equal  ${fail_count}  0
-    ...  msg=Patch operation failed ${fail_count} times in ${verify_count} attempts
+    Should Be Equal As Integers  ${fail_count}  ${0}
+    ...  msg=Patch operation failed ${fail_count} times in ${verify_count} attempts; fails at iterations ${fail_list}

 Verify User Cannot Login After 5 Non-Logged In Sessions
@@ -100,11 +101,11 @@ Flood Post Without Auth Token And Check Stability Of BMC
     [Documentation]  Flood post method without auth token and check BMC stability.
     [Tags]  Flood_Post_Without_Auth_Token_And_Check_Stability_Of_BMC

-    @{status_list}=  Create List
+    @{fail_list}=  Create List
     ${user_info}=  Create Dictionary
     ...  UserName=test_user  Password=TestPwd123  RoleId=Operator  Enabled=${True}

-    FOR  ${i}  IN RANGE  ${1}  ${iterations}
+    FOR  ${i}  IN RANGE  ${1}  ${iterations} + 1
         Log To Console  ${i}th iteration
         Run Keyword And Ignore Error
         ...  Redfish.Post   /redfish/v1/AccountService/Accounts/  body=&{user_info}
@@ -112,13 +113,13 @@ Flood Post Without Auth Token And Check Stability Of BMC
         # Every 100th iteration, check BMC allows post with auth token.
         ${status}=  Run Keyword If  ${i} % 100 == 0  Run Keyword And Return Status
         ...  Login And Create User
-        Run Keyword If  ${status} == False  Append To List  ${status_list}  ${status}
+        Run Keyword If  ${status} == False  Append To List  ${fail_list}  ${i}
     END
     ${verify_count}=  Evaluate  ${iterations}/100
-    ${fail_count}=  Get Length  ${status_list}
+    ${fail_count}=  Get Length  ${fail_list}

-    Should Be Equal  ${fail_count}  0
-    ...  msg=Post operation failed ${fail_count} times in ${verify_count} attempts
+    Should Be Equal As Integers  ${fail_count}  ${0}
+    ...  msg=Post operation failed ${fail_count} times in ${verify_count} attempts; fails at iterations ${fail_list}

 Make Large Number Of Wrong SSH Login Attempts And Check Stability
@@ -161,7 +162,8 @@ Test Stability On Large Number Of Wrong Login Attempts To GUI

     Go To  ${bmc_url}

-    FOR  ${i}  IN RANGE  ${1}  ${iterations}
+    Printn
+    FOR  ${i}  IN RANGE  ${1}  ${iterations} + 1
         Log To Console  ${i}th login
         Run Keyword And Ignore Error  Login to GUI With Incorrect Credentials
gkeishin commented 2 years ago

@prkatti1 can u take a look

gkeishin commented 2 years ago

Thank you @generatz for the feedback.. will look into it..

gkeishin commented 2 years ago

Changes : https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-test-automation/+/52701

update extra improvement along with it