search

canonical / testflinger

https://testflinger.readthedocs.io/en/latest/
GNU General Public License v3.0
9 stars 13 forks source link

Add firmware_update phase support for HPE Gen10/Gen11/RL machines #255

Open nancyc12 opened 2 months ago

nancyc12 commented 2 months ago

Description

Based on CR033 - Firmware updates for regression testing, add firmware_update phase support for HPE server machines utilising ilorest (apt install from HPE repository) and the firmware files downloaded from HPE firmware repository.

HPE firmware repository index files (fwrepo.json) on HPE firmware repository have been updated to support multiple targets (hardware components) per firmware file, after communicating with HPE.

This PR also includes these changes:

Resolved issues

N/A

Documentation

test-phases.rst updated accordingly.

Tests

2024-04-16 08:41:14,964 doubletusk INFO: DEVICE CONNECTOR: BEGIN firmware_update 2024-04-16 08:41:14,965 doubletusk INFO: DEVICE CONNECTOR: check and wait for 60s until 10.245.130.29 is SSHable 2024-04-16 08:41:20,970 doubletusk INFO: DEVICE CONNECTOR: 10.245.130.29 is SSHable after 6s 2024-04-16 08:41:21,528 doubletusk INFO: DEVICE CONNECTOR: 10.245.130.29 is a HPE HPEDevice 2024-04-16 08:41:30,833 doubletusk INFO: DEVICE CONNECTOR: successfully installed RESTful Interface Tool 4.2.0.0 2024-04-16 08:42:10,794 doubletusk INFO: DEVICE CONNECTOR: HPE firmware repository: https://downloads.linux.hpe.com/SDR/repo/fwpp-gen10/ 2024-04-16 08:42:17,356 doubletusk INFO: DEVICE CONNECTOR: Available firmware options: ['2023.09.00.04', '2023.09.00.03', '2023.09.00.02', '2023.09.00.01', '2023.09.00.00', '2023.03.00.00', '2022.09.1', '2022.03.1', '2022.03.0_supspp_rhel9.0_x86_64', '2022.03.0', '2021.10.0', '2021.04.0', '2020.09.0_hotfix_1', '2020.09.0', '2020.03.2', '2020.03.0_hotfix_1', '2020.03.0', '2019.12.0_hotfix_1', '2019.12.0'] 2024-04-16 08:42:22,543 doubletusk INFO: DEVICE CONNECTOR: start flashing all firmware with files in SPP 2023.09.00.04 2024-04-16 08:42:26,814 doubletusk INFO: DEVICE CONNECTOR: firmware updatable on HPE machine: [{'Firmware Name': 'System ROM', 'Firmware Version': 'U30 v3.00 (10/19/2023)', 'Location': 'System Board', 'Fwpkg Available': {'2023.09.00.04': {'file': 'U30_3.00_10_19_2023.fwpkg', 'date': '20231027', 'description': 'ROM Flash Firmware Package - HPE ProLiant DL380 Gen10 (U30) Servers', 'deviceclass': 'aa148d2e-6e09-453e-bc6f-63baa5f5ccc4', 'minimum_active_version': 'null', 'reboot_required': 'yes', 'target': ['00000000-0000-0000-0000-000000000205'], 'version': '3.00_10-19-2023'}}, 'Update Order': 0, 'Device Class': 'aa148d2e-6e09-453e-bc6f-63baa5f5ccc4', 'Target ID': ['00000000-0000-0000-0000-000000000205', '00000000-0000-0000-0000-000001553330']}, {'Firmware Name': 'Server Platform Services (SPS) Firmware', 'Firmware Version': '4.1.5.2', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 1, 'Device Class': 'b34e5677-21dc-45d3-872b-42f76fee9053', 'Target ID': ['00000000-0000-0000-0000-000000000205', 'a6b1a447-382a-5a4f-3c10-86800a000101']}, {'Firmware Name': 'Server Platform Services (SPS) Descriptor', 'Firmware Version': '1.2 0', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 1, 'Device Class': '3f30a329-d03c-46b8-8f0e-9567fad4ea9f', 'Target ID': ['00000000-0000-0000-0000-000000000205']}, {'Firmware Name': 'Innovation Engine (IE) Firmware', 'Firmware Version': '0.2.3.0', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 2, 'Device Class': 'c734e171-8721-48c9-9ed6-d5bc7da5ef8d', 'Target ID': ['00000000-0000-0000-0000-000000000205', 'a6b1a447-382a-5a4f-3c10-3c100a000303']}, {'Firmware Name': 'iLO 5', 'Firmware Version': '3.00 Dec 14 2023', 'Location': 'System Board', 'Fwpkg Available': {'2023.09.00.04': {'file': 'ilo5_301.fwpkg', 'date': '20240123', 'description': 'Online ROM Flash Firmware Package - HPE Integrated Lights-Out 5', 'deviceclass': '2f317b9d-c9e3-4d76-bff6-b9d0d085a952', 'minimum_active_version': 'null', 'reboot_required': 'no', 'target': ['9c057a7a-0d48-4260-b033-558f22ef7c6a', 'c5b14abd-6d2f-41d7-b665-0447dfdac3a7', 'c0bcf2b9-1141-49af-aab8-c73791f0349c'], 'version': '3.01'}}, 'Update Order': 3, 'Device Class': '2f317b9d-c9e3-4d76-bff6-b9d0d085a952', 'Target ID': ['4764a662-b342-4fc7-9ce9-258c5d99e815', 'c0bcf2b9-1141-49af-aab8-c73791f0349c']}, {'Firmware Name': 'Redundant System ROM', 'Firmware Version': 'U30 v2.90 (07/20/2023)', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 4, 'Device Class': '', 'Target ID': ''}, {'Firmware Name': 'Intelligent Provisioning', 'Firmware Version': '3.50.100', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 4, 'Device Class': '', 'Target ID': ''}, {'Firmware Name': 'Power Management Controller FW Bootloader', 'Firmware Version': '1.1', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 4, 'Device Class': '', 'Target ID': ''}, {'Firmware Name': 'Intelligent Platform Abstraction Data', 'Firmware Version': '16.5.0 Build 53', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': 'b8f46d06-85db-465c-94fb-d106e61378ed', 'Target ID': ['00000000-0000-0000-0000-000000000205', '00000000-0000-0000-0000-000001553330']}, {'Firmware Name': 'System Programmable Logic Device', 'Firmware Version': '0x31', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': 'b1ad439a-9dd1-41c1-a496-2da9313f1f07', 'Target ID': ['00000000-0000-0000-0000-000000000205']}, {'Firmware Name': 'Power Management Controller Firmware', 'Firmware Version': '1.1.0', 'Location': 'System Board', 'Fwpkg Available': {'2023.09.00.04': {'file': 'PICGen10-1.1.0-1.fwpkg', 'date': '20231022', 'description': 'ROM Flash Firmware Package - Advanced Power Capping Microcontroller Firmware for HPE Gen10 and Gen10 Plus Servers', 'deviceclass': '9e48a28a-586c-4519-8405-a04f84e27f0f', 'minimum_active_version': 'null', 'reboot_required': 'yes', 'target': ['00000000-0000-0000-0000-000000504d05'], 'version': '1.1.0'}}, 'Update Order': 5, 'Device Class': '9e48a28a-586c-4519-8405-a04f84e27f0f', 'Target ID': ['00000000-0000-0000-0000-000000000205', '00000000-0000-0000-0000-000000504d05']}, {'Firmware Name': 'NVMe Backplane Firmware', 'Firmware Version': '1.24', 'Location': 'System Board', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': '3653aa90-7089-453a-899c-792827a50d67', 'Target ID': ['00000000-0000-0000-0000-000000000205', '00000000-0000-0000-0000-000049535320']}, {'Firmware Name': 'Power Supply Firmware', 'Firmware Version': '2.00', 'Location': 'Bay 1', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': '6bb86077-0275-4614-aae1-86618e8b1c27', 'Target ID': ['0302b505-0002-0000-0000-0cf38db966ea']}, {'Firmware Name': 'Power Supply Firmware', 'Firmware Version': '2.00', 'Location': 'Bay 2', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': '6bb86077-0275-4614-aae1-86618e8b1c27', 'Target ID': ['0302b505-0002-0000-0000-0cf38db966ea']}, {'Firmware Name': 'HPE Eth 10/25Gb 2p 640FLR-SFP28 Adptr', 'Firmware Version': '14.32.1010', 'Location': 'Embedded ALOM', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-15b3-1015159000d3']}, {'Firmware Name': 'HPE Smart Array E208i-a SR Gen10', 'Firmware Version': '6.22', 'Location': 'Embedded RAID', 'Fwpkg Available': {'2023.09.00.04': {'file': 'HPE_SR_Gen10_6.52_A.fwpkg', 'date': '20231031', 'description': 'Firmware Package - HPE Smart Array P408i-p, P408e-p, P408i-a, P408i-c, E208i-p, E208e-p, E208i-c, E208i-a, P408e-m, P204i-c, P416ie-m and P816i-a SR Gen10 and Gen11 controllers', 'deviceclass': '79f0c163-0c13-4662-9dea-09235fef90cb', 'minimum_active_version': '5.61', 'reboot_required': 'yes', 'target': ['a6b1a447-382a-5a4f-9005-028f103c1100', 'a6b1a447-382a-5a4f-9005-028f103c1101', 'a6b1a447-382a-5a4f-9005-028f103c0601', 'a6b1a447-382a-5a4f-9005-028f103c0602', 'a6b1a447-382a-5a4f-9005-028f103c0603', 'a6b1a447-382a-5a4f-9005-028f103c0600', 'a6b1a447-382a-5a4f-9005-028f103c0700', 'a6b1a447-382a-5a4f-9005-028f103c0651', 'a6b1a447-382a-5a4f-9005-028f103c0654', 'a6b1a447-382a-5a4f-9005-028f103c0652', 'a6b1a447-382a-5a4f-9005-028f103c0650', 'a6b1a447-382a-5a4f-9005-028f15900382', 'a6b1a447-382a-5a4f-9005-028f15900383'], 'version': '6.52'}}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-9005-028f103c0654']}, {'Firmware Name': 'NVIDIA Tesla T4', 'Firmware Version': '90.04.96.00.01', 'Location': 'PCI-E Slot 2', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-10de-1eb810de12a2']}, {'Firmware Name': 'NVIDIA Tesla T4', 'Firmware Version': '90.04.96.00.01', 'Location': 'PCI-E Slot 3', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-10de-1eb810de12a2']}, {'Firmware Name': 'NVIDIA Tesla T4', 'Firmware Version': '90.04.96.00.01', 'Location': 'PCI-E Slot 5', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-10de-1eb810de12a2']}, {'Firmware Name': 'NVIDIA Tesla T4', 'Firmware Version': '90.04.96.00.01', 'Location': 'PCI-E Slot 6', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-10de-1eb810de12a2']}, {'Firmware Name': 'Embedded Video Controller', 'Firmware Version': '2.5', 'Location': 'Embedded Device', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': None, 'Target ID': ['a6b1a447-382a-5a4f-102b-0538159000e4']}, {'Firmware Name': 'NVMe Drive', 'Firmware Version': 'EDA5302Q', 'Location': 'NVMe Drive Port 5B Box 1 Bay 1', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': 'f6de0320-2e0f-489a-b238-6dd8ae7c3811', 'Target ID': ['532340a5-6d61-7573-6e67-20532a9e3acf']}, {'Firmware Name': 'NVMe Drive', 'Firmware Version': 'EDA5302Q', 'Location': 'NVMe Drive Port 5B Box 1 Bay 2', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': 'f6de0320-2e0f-489a-b238-6dd8ae7c3811', 'Target ID': ['532340a5-6d61-7573-6e67-20532a9e3acf']}, {'Firmware Name': 'HPE 2SFF NVMe/SAS/uFF Backplane', 'Firmware Version': '1.24', 'Location': 'Box=1', 'Fwpkg Available': {}, 'Update Order': 5, 'Device Class': '4149ebb7-d64b-4654-b679-4280f349ea50', 'Target ID': ['9675b54e-1c96-44c3-a7a1-901500000000']}] 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: [System ROM] no update is needed, already U30 v3.00 (10/19/2023) 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: [iLO 5] update current firmware 3.00 Dec 14 2023 to 3.01 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: ilo5_301.fwpkg is already downloaded 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: [Power Management Controller Firmware] no update is needed, already 1.1.0 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: [HPE Smart Array E208i-a SR Gen10] update current firmware 6.22 to 6.52 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: HPE_SR_Gen10_6.52_A.fwpkg is already downloaded 2024-04-16 08:42:29,623 doubletusk INFO: DEVICE CONNECTOR: check and clear iLO taskqueue 2024-04-16 08:42:50,030 doubletusk INFO: DEVICE CONNECTOR: start flashing ilo5_301.fwpkg 2024-04-16 08:46:44,093 doubletusk INFO: DEVICE CONNECTOR: Uploading firmware: ilo5_301.fwpkg Successfully checked 'ilo5_301.fwpkg'. Uploading component ilo5_301.fwpkg. [200] The operation completed successfully. Component ilo5_301.fwpkg uploaded successfully. Waiting for iLO UpdateService to finish processing the component 00 hour(s) 04 minute(s) 38 second(s) Firmware has successfully been flashed. iLO will reboot to complete flashing. Session will be terminated. 2024-04-16 08:46:44,094 doubletusk INFO: DEVICE CONNECTOR: wait until iLO complete reboot 2024-04-16 08:47:43,454 doubletusk INFO: DEVICE CONNECTOR: start flashing HPE_SR_Gen10_6.52_A.fwpkg 2024-04-16 08:49:47,111 doubletusk INFO: DEVICE CONNECTOR: Uploading firmware: HPE_SR_Gen10_6.52_A.fwpkg Successfully checked 'HPE_SR_Gen10_6.52_A.fwpkg'. Uploading component HPE_SR_Gen10_6.52_A.fwpkg. [200] The operation completed successfully. Component HPE_SR_Gen10_6.52_A.fwpkg uploaded successfully. Waiting for iLO UpdateService to finish processing the component 00 hour(s) 02 minute(s) 44 second(s) Firmware has successfully been flashed and a reboot is required for this firmware to take effect. 2024-04-16 08:49:47,113 doubletusk INFO: DEVICE CONNECTOR: check iLO access 2024-04-16 08:50:09,507 doubletusk INFO: DEVICE CONNECTOR: iLO connection checked 2024-04-16 08:50:09,507 doubletusk INFO: DEVICE CONNECTOR: reboot DUT 2024-04-16 08:59:02,384 doubletusk INFO: DEVICE CONNECTOR: HPE machine reaches FinishedPost after 512s 2024-04-16 08:59:12,393 doubletusk INFO: DEVICE CONNECTOR: check and wait for 1800s until 10.245.130.29 is SSHable 2024-04-16 08:59:18,463 doubletusk INFO: DEVICE CONNECTOR: 10.245.130.29 is SSHable after 6s 2024-04-16 08:59:35,577 doubletusk INFO: DEVICE CONNECTOR: [iLO 5] firmware flashed 3.00 Dec 14 2023 → 3.01 Jan 23 2024 2024-04-16 08:59:35,578 doubletusk INFO: DEVICE CONNECTOR: [HPE Smart Array E208i-a SR Gen10] firmware flashed 6.22 → 6.52 2024-04-16 08:59:41,533 doubletusk INFO: DEVICE CONNECTOR: END firmware_update