datacenter / ACI-Pre-Upgrade-Validation-Script

A script to run validations to detect potential issues that may cause an ACI fabric upgrade to fail
https://datacenter.github.io/ACI-Pre-Upgrade-Validation-Script/
Apache License 2.0
41 stars 26 forks source link

APIC Target version image and MD5 hash fails with 'failed to login to host' #133

Closed ian-baxter-arm closed 2 months ago

ian-baxter-arm commented 3 months ago

Describe the bug

Test 1 'APIC Target version image and MD5 hash' fails with the error ls command via ssh failed due to:failed to login to host on all hosts.

This happens because there is no response to the SSH command passed to pexpect (lines 185-186) resulting in a match of 'eof' in the connection class login function. As it doesn't match any of the expected matches we get the default error on line 302.

After testing the SSH command on the APIC CLI I found you do get a response when you remove -o HostKeyAlgorithms=+ssh-dss which is added on line 183. After removing this line from the script the test completed successfully.

Line references in file aci-preupgrade-validation-script.py on commit 6c16ae99752a0c54ad158e79a6a11104068a2813

Script output

[Check  1/64] APIC Target version image and MD5 hash... 
              Checking apic-srv-1......                                                                              ERROR !!
              Checking apic-srv-2......                                                                              ERROR !!
              Checking apic-srv-3......                                                                              ERROR !!
                                                                                                                  FAIL - UPGRADE FAILURE!!
  APIC                     Firmware  md5sum  Failure                                                   Recommended Action
  ----                     --------  ------  -------                                                   ------------------
  apic-srv-1               -         -       ls command via ssh failed due to:failed to login to host  -
  apic-srv-3               -         -       ls command via ssh failed due to:failed to login to host  -
  apic-srv-2               -         -       ls command via ssh failed due to:failed to login to host  -

Server information

admin@apic-svr-1:~> cat /etc/*release
RFS_VERSION=2.32.10
CENTOS_APIC_VERSION=7.9.2009
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
admin@apic-svr-1:~> ssh -V
OpenSSH_8.9p1, CiscoSSL 1.0.2k.6.1.188-fips

To Reproduce Steps to reproduce the behavior such as:

  1. Run the script (v2.1.0) on version 5.2(8h) with the target version 5.2(8i)

Command with no response:

ssh  -o StrictHostKeyChecking=no -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -o HostKeyAlgorithms=+ssh-dss admin@[IP removed] -p 22

Command with response

ssh  -o StrictHostKeyChecking=no -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null admin@[IP removed] -p 22

Expected behavior The test should have been able to log in and test the files.

ian-baxter-arm commented 3 months ago

Another APIC we have not affected by this bug. Seems there is a difference in the version of OpenSSH.

ACI version 5.2(7f)

admin@apic-srv-1:~> cat /etc/*release
RFS_VERSION=2.32.10
CENTOS_APIC_VERSION=7.9.2009
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)
admin@apic-srv-1:~> ssh -V
OpenSSH_7.4p1, CiscoSSL 1.0.2k.6.1.188-fips