search

bb-Ricardo / check_redfish

A monitoring/inventory plugin to check components and health status of systems which support Redfish. It will also create a inventory of all components of a system.
MIT License
113 stars 34 forks source link

object of type 'NoneType' has no len() #100

Closed Eldiabolo21 closed 1 year ago

Eldiabolo21 commented 1 year ago

Hi Ricardo,

next issue ;) When running inventory against a Supermicro X12SPi-TF I get: 

Traceback (most recent call last):
  File "./check_redfish.py", line 172, in <module>
    if any(x in args.requested_query for x in ['nic', 'all']):      get_network_interfaces(plugin)
  File "/home/fclauss/check_redfish/cr_module/nic.py", line 23, in get_network_interfaces
    get_system_nics(plugin_object, system)
  File "/home/fclauss/check_redfish/cr_module/nic.py", line 373, in get_system_nics
    num_ports = len(network_ports)
TypeError: object of type 'NoneType' has no len()

Command: ./check_redfish.py -H 10.33.32.82 -u admin -p XXXXXXXXX --inventory --all --inventory_file host.json --inventory_id 2092

Tried with the current masterbranch as well as next-release.

bb-Ricardo commented 1 year ago

I pushed an update to next-release. Can you please test it and see if it fixes the issue?

thank you

Eldiabolo21 commented 1 year ago

This is the output. Is that expected? 

./check_redfish.py -H 10.33.32.82 -u admin -p XXXXXXXXX --inventory --all --inventory_file host.json --inventory_id 2092
Object id '1:1' for 'NetworkPort' already used
Object id '1' for 'Firmware' already used
Object id '2' for 'Firmware' already used
[OK]: Successfully written inventory file but inventory data might be incomplete

Json file itself looks good to me!

bb-Ricardo commented 1 year ago

This is not meant to be happening. Do you mind sending me a MockUp for this server and I can have a look why this is happening. Then I could also try to fix the parsing for this type of hosts.

Thank you

Eldiabolo21 commented 1 year ago

Do you mean with https://github.com/DMTF/Redfish-Mockup-Creator ?

Will the mockup contain also the data in the BMC or just the structure of the endpoints? If its the former, i'll need to talk to our SEC-Department.

bb-Ricardo commented 1 year ago

Hi, yes this project is correct.

And yes it will include all the data of this servers bmc.

Eldiabolo21 commented 1 year ago

Okay, on it, might take some time.

bb-Ricardo commented 1 year ago

Hey, any updates on this? Would be great if we could fix this issue.

bb-Ricardo commented 1 year ago

Hey @Eldiabolo21, any updates on this?

Eldiabolo21 commented 1 year ago

Hi Ricardo, sorry for the late reply, unfortuntely I will be leaving my current company so I can't work on this issue anymore. I wil try to hand it over to someone else. I will reply soon how to proceed.

Kegeruneku commented 1 year ago

Hello @bb-Ricardo , got the same issue so I'll gladly help you on this one :-) (running on a Supermicro SSG-620P-E1CR24H server with a big load of disks inside).

Output when querying --all:

Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:1' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:2' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:3' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:4' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:5' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:6' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:7' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:8' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:9' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:10' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:11' for 'PhysicalDrive' already used
Object id 'NVMeSSD:1:0' for 'PhysicalDrive' already used
Object id 'NVMeSSD:1:1' for 'PhysicalDrive' already used
Object id '1:1' for 'NetworkPort' already used

Will try to mockup the server ASAP.

Kegeruneku commented 1 year ago

... aaaaand done, mockup ready :-) where can I reach you to send it ?

bb-Ricardo commented 1 year ago

great, here would be great: https://github.com/bb-Ricardo/check_redfish/blob/d5c4ee96d24266c6b172c31758682c187a2be8ad/check_redfish.py#L21

Thank you very much

Kegeruneku commented 1 year ago

And here goes !

I notice something interesting in the output:

Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
(...)
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
(...)
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
Object id 'HA-RAID:1:0' for 'PhysicalDrive' already used
(...)

It seems the same object ID is queried at least four times in a row, possibly due to beeing stacked more than once.

I did not look at the code yet, but is it possible that the script gets the object id's from multiple sources that might outout the same id's ?

bb-Ricardo commented 1 year ago

Thank you, I got the mockup.

Will have a look what's going on here. could be that the controller IDs are not unique. But let's see.

bb-Ricardo commented 1 year ago

I had a look and can see the issue.

Not quiete sure how to solve it elegant. Supermicro is not helping much with their implementation. The Disk entry does not contain a reference to which controller and enclosure this drive belongs to except for the path. Which means this would end up in a sepcific SuperMicro implementention which might change again with the next release of their BMC firmware.

Also all the drives seem to be listed again under "SimpleStorage", But then there seem to be even more. And SimpleStorage does not contain information about location or a serial number which makes it impossible to cross reference to filter out duplicate entries.

bb-Ricardo commented 1 year ago

Hi @Kegeruneku, @Eldiabolo21:

I found out that this seems a Supermicro specific issue and their not quite optimal BMC firmware implementation (using IDs more then one, structure of presenting storage) I added some mitigation to prevent the errors. But was not able to remove the duplicate reported storage.

Maybe Supermicro fixes this in future firmware releases.

I pushed a new commit to next-release. Would you be able to test and see if that fixes your issue?

Thank you.

Kegeruneku commented 1 year ago

Hello @bb-Ricardo ! Looking good with the patch, no Object id 'HA-RAID:X:X' for 'PhysicalDrive' already used before the final output 👍

Kegeruneku commented 1 year ago

Sample output:

# ./check_redfish.py -H XX.XX.XX.XX -u ADMIN -p XXXXXXXX
 --all
[OK]: BMC: ASPEED (Firmware: 01.01.24) and all nics are in 'OK' state.
[OK]: Chassi 1 : All fans (7) are in good condition
[OK]: Chassi NVMeSSD.0.Group.0.StorageBackplane : Chassi has no fans installed/reported
[OK]: Chassi NVMeSSD.1.Group.1.StorageBackplane : Chassi has no fans installed/reported
[OK]: Found 19 firmware entries. Use '--detailed' option to display them.
[OK]: Manager Event Log contains 349 OK entries. Most recent notable: [OK]: 2023-05-26T13:14:22Z: [MEL-0133] Redfish session was created successfully.
[OK]: All memory modules (Total 256GB) are in good condition
[OK]: All network adapter (1) and ports (5) are in good condition
[OK]: Chassi 1 : All power supplies (2) are in good condition and Power redundancy 1 status is: Enabled and 20 Voltages are OK
[OK]: Chassi NVMeSSD.0.Group.0.StorageBackplane : Chassi has no power supplies installed/reported
[OK]: Chassi NVMeSSD.1.Group.1.StorageBackplane : Chassi has no power supplies installed/reported
[OK]: All processors (2) are in good condition
[OK]: All storage controllers (3), volumes (48) and disk drives (56) are in good condition
[OK]: No System log entries found in '/redfish/v1/Systems/1/LogServices/Log1/Entries'.
[OK]: INFO: Supermicro SSG-620P-E1CR24H (CPU: 2, MEM: 256GB) - BIOS: 1.4 - Serial: XXXXXXXXXXXX - Power: On - Name: NOT SET
[OK]: Chassi 1 : All temp sensors (24) are in good condition
[OK]: Chassi NVMeSSD.0.Group.0.StorageBackplane : Chassi has no temp sensors installed/reported
[OK]: Chassi NVMeSSD.1.Group.1.StorageBackplane : Chassi has no temp sensors installed/reported|'ps_1.1'=300 'ps_1.2'=268 'voltage_1.Vcpu1VCCIN'=1.804 'voltage_1.Vcpu1VCCIO'=1.046 'voltage_1.Vcpu2VCCIN'=1.802 'voltage_1.Vcpu2VCCIO'=1.052 'voltage_1.Vp1ABCD'=1.229 'voltage_1.Vp1EFGH'=1.238 'voltage_1.Vp2ABCD'=1.23 'voltage_1.Vp2EFGH'=1.224 'voltage_1.12V'=12.237 'voltage_1.5VCC'=5.07 'voltage_1.3.3VCC'=3.322 'voltage_1.5VSB'=5.082 'voltage_1.3.3VSB'=3.346 'voltage_1.1.8V_PCH'=1.82 'voltage_1.PVNN_PCH'=1.012 'voltage_1.1.05V_PCH'=1.058 'voltage_1.2.5V_BMC'=2.527 'voltage_1.1.8V_BMC'=1.816 'voltage_1.1.2V_BMC'=1.204 'voltage_1.1.0V_BMC'=0.999 'power_control_System_Power_Control'=560.0 'temp_1.CPU1_Temp'=68.0;;98 'temp_1.CPU2_Temp'=60.0;;98 'temp_1.PCH_Temp'=42.0;;90 'temp_1.System_Temp'=38.0;;85 'temp_1.Peripheral_Temp'=46.0;;85 'temp_1.CPU1_VRMIN_Temp'=54.0;;100 'temp_1.CPU1_VRMIO_Temp'=54.0;;100 'temp_1.CPU2_VRMIN_Temp'=48.0;;100 'temp_1.CPU2_VRMIO_Temp'=51.0;;100 'temp_1.P1_VRMABCD_Temp'=45.0;;100 'temp_1.P1_VRMEFGH_Temp'=48.0;;100 'temp_1.P2_VRMABCD_Temp'=39.0;;100 'temp_1.P2_VRMEFGH_Temp'=42.0;;100 'temp_1.P1_DIMMA~D_Temp'=48.0;;85 'temp_1.P1_DIMME~H_Temp'=45.0;;85 'temp_1.P2_DIMMA~D_Temp'=44.0;;85 'temp_1.P2_DIMME~H_Temp'=42.0;;85 'temp_1.Expander_Temp'=54.0;;100 'temp_1.AOC_SAS1_Temp'=47.0;;105 'temp_1.AOC_SAS2_Temp'=47.0;;105 'temp_1.HDD_Temp'=36.0;;55 'temp_1.AIOM_NIC1_Temp'=53.0;;100 'temp_1.U2_SSDA_Temp'=48.0;;70 'temp_1.U2_SSDB_Temp'=42.0;;70 'Fan_1.1'=9940;; 'Fan_1.2'=10080;; 'Fan_1.3'=9940;; 'Fan_1.4'=10220;; 'Fan_1.5'=10220;; 'Fan_1.A'=13720;; 'Fan_1.B'=13860;;
bb-Ricardo commented 1 year ago

great, thank you for testing, then I will close this issue.