Security.Test Bmc Ssh Security.Verify BMC SSH Weak Cipher And Algorithm test fails, but returns PASS

The test is designed to expect the ssh command to fail because login permission is denied, but if the ssh command fails for other reasons the test case still passes, even though it has not fulfilled the test case goal of checking the ciphers.

In my working environment (see below), the test case fails because the ssh command line doesn't contain '-o' in front of each of the second and third ssh option arguments, as demonstrated below. failing code:

    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 UserKnownHostsFile=/dev/null
    ...  StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1

remediated code:

    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 -o UserKnownHostsFile=/dev/null
    ...  -o StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1

I observed this failure on Ubuntu 18.04 with OpenSSH 7.6p1 / OpenSSL 1.0.2 and Python 3.6.9.

I propose improving the test case by first verifying that the command returns the kind of information desired. In my environment the remediated code returns the error 'Permission denied' since the ssh attempt requests no password prompt ('-o NumberOfPasswordPrompts=0). My refactored code expects that string to be returned, and returns SKIP if that criterion is not satisfied.

Here is the git diff showing how I have refactored the code:

diff --git a/security/test_bmc_ssh_security.robot b/security/test_bmc_ssh_security.robot
index fae06281..c70a3972 100644
--- a/security/test_bmc_ssh_security.robot
+++ b/security/test_bmc_ssh_security.robot
@@ -41,6 +41,8 @@ Verify BMC SSH Weak Cipher And Algorithm
     # debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,
     #         hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,
     #         umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+    #         ...
+    #         ... Permission denied (publickey,password).

     # Example of weak algorithms to check:
     # - encryption: triple-DES ("DES-CBC3").
@@ -49,7 +51,16 @@ Verify BMC SSH Weak Cipher And Algorithm
     # - KEX: diffie-hellman-group1(any) , (any) SHA1

     Printn
-    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 UserKnownHostsFile=/dev/null
-    ...  StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1
+    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 -o UserKnownHostsFile=/dev/null
+    ...    -o StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1
+
+    ${rc}  ${std_err}=  Shell Cmd  ! ${ssh_cmd_buf}
+    Log  std_err=${std_err}  console=yes
+    Log  rc=${rc} console=yes
+
+    ${has_it}=  Run Keyword And Return Status  Should Contain  ${std_err}  Permission denied
+    Skip If  not ${has_it}
+    ...  Skipping test case since response is not as expected
+    
     Shell Cmd  ! ${ssh_cmd_buf} | egrep -- "${weak_key_regex}"
     Shell Cmd  ! ${ssh_cmd_buf} | egrep -- "${mac_key_regex}"

openbmc / openbmc-test-automation

Security.Test Bmc Ssh Security.Verify BMC SSH Weak Cipher And Algorithm test fails, but returns PASS #2176