rcbops / openstack-ops

Tasks and Scripts written by the RPC Support team for use in RPC environments
Apache License 2.0
14 stars 24 forks source link

HPE Firmware update script has issues with controller check when multiple controllers exsit. #212

Closed rax-rstark closed 9 months ago

rax-rstark commented 9 months ago

The HPE firmware update script check for storage controller version does not properly check hosts containing multiple storage controllers.

694003-cinderSSD02 is a cinder node with 3 storage controllers; Embedded, Slot 1, and Slot 3. The update_hp_firmware.py script runs the following check to determine if a storage controller firmware update is needed:

# ssacli controller all show config detail | grep -i firmware\ version | cut -d: -f2 | tr -d ' '| head -1
7.20-0

when run without head, we can see the existence of multiple controller versions:

# ssacli controller all show config detail | grep -i firmware\ version | cut -d: -f2 | tr -d ' '
7.20-0
6.88-0
7.20-0

Can we update the logic of this script to accommodate checking for multiple controller versions?

jklint commented 9 months ago

Just a thought, if we sorted the output, the lowest version would always be first and meet the condition of needing an upgrade should it be below the checked version.

# ssacli controller all show config detail | grep -i firmware\ version | cut -d: -f2 | tr -d ' '| sort -n | head -1

shahzaib-bhatia commented 9 months ago

This info's also available through sysfs at /sys/class/scsi_host/host*/firmware_revision f.ex: # cat /sys/class/scsi_host/host*/firmware_revision | sort -n | head -n1 will give you the lowest version.

rax-rstark commented 9 months ago

Not having a dependency on ssacli commands would be nice as they add unneeded latency in check response.

BjoernT commented 9 months ago

I still favor ssacli as we can narrow the controllers to supported only and don't have to deal with different vendors. So I just made the small change to sort like proposed. Check it out and let me know

rax-rstark commented 9 months ago

As luck would have it, we have a cinder node in C2 that has 3 controllers, 2 at current version and one that is very, very out of date - 698735-cinderSSD04.

# ssacli controller all show config detail | grep -i firmware\ version | cut -d: -f2 | tr -d ' '
7.20-0
7.20-0
4.52-0

Using the modified update script in check mode shows the expected result for storage controller upgrade:

root@698735-cinderssd04:~# python /root/update_hp_firmware.py
Verified supported hardware: ProLiant DL380 Gen9
Checking for intial pre-requisites...

*********************************************
PERFORMING DRY RUN. NO CHANGES WILL BE DONE.
*********************************************

Verifying current NIC version: Needs update ( -> 5719-v1.46NCSIv1.5.1.0)

Verifying current SYSTEM version: Needs update (01/12/2023 -> 09/21/2023)

Verifying current RAID version: Needs update (4.52-0 -> 7.20)

Verifying current ILO version: Already current (2.82). Nothing to do.

Verifying current Intel 10G NIC Versions:
 p2p2 already current. Nothing to do.
 p2p1 already current. Nothing to do.
 em50 needs update (800005B6 -> 80000636)
 em49 needs update (800005B6 -> 80000636)

SUMMARY: Needs Update

LGTM!