search

mrlhansen / idrac_exporter

Simple Redfish (iDRAC, iLO, XClarity) exporter for Prometheus
MIT License
131 stars 40 forks source link

HPE server cannot obtain memory, disk, network and other information #57

Closed ilanni2460 closed 8 months ago

ilanni2460 commented 8 months ago

HPE Server iLO 4 ProLiant DL580 Gen9 cannot obtain memory, disk, network and other information

idrac_exporter debug log:

2024-02-22T19:19:47.075 DEBUG Querying url "https://10.10.190.32/redfish/v1"
2024-02-22T19:19:48.659 DEBUG Querying url "https://10.10.190.32/redfish/v1/Systems/"
2024-02-22T19:19:49.583 DEBUG Querying url "https://10.10.190.32/redfish/v1/Chassis/"
2024-02-22T19:19:50.467 DEBUG Querying url "https://10.10.190.32/redfish/v1/Chassis/1/"
2024-02-22T19:19:51.328 DEBUG Querying url "https://10.10.190.32/redfish/v1/Systems/1/"
2024-02-22T19:19:52.783 DEBUG Collecting metrics for host 10.10.190.32

NetworkInface:

curl -k https://10.10.190.32/redfish/v1/Systems/1/EthernetInterfaces/4/ 

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/EthernetInterfaces/Members/$entity",
  "@odata.id": "/redfish/v1/Systems/1/EthernetInterfaces/4/",
  "@odata.type": "#EthernetInterface.1.0.0.EthernetInterface",
  "FullDuplex": false,
  "IPv4Addresses": [
    {
      "Address": null
    }
  ],
  "IPv6Addresses": [
    {
      "Address": null
    }
  ],
  "Id": "4",
  "MacAddress": "3C:A8:2A:ED:E9:DB",
  "Name": "System Ethernet Interface",
  "SpeedMbps": null,
  "Status": {
    "Health": "OK",
    "State": "Enabled"
  },
  "Type": "EthernetNetworkInterface.0.92.0",
  "UEFIDevicePath": "PciRoot(0x0)/Pci(0x2,0x2)/Pci(0x0,0x0)",
  "links": {
    "self": {
      "href": "/redfish/v1/Systems/1/EthernetInterfaces/4/"
    }
  }
}

1

Memory:

curl -k https://10.10.190.32/redfish/v1/Systems/1/Memory/

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/Memory",
  "@odata.id": "/redfish/v1/Systems/1/Memory/",
  "@odata.type": "#HpMemoryCollection.HpMemoryCollection",
  "Description": "Memory DIMM Collection",
  "MemberType": "HpMemory.1",
  "Members": [
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board1dimm4/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board1dimm9/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board3dimm4/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board3dimm9/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board5dimm4/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board5dimm9/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board7dimm4/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/Memory/board7dimm9/"
    }
  ],
  "Members@odata.count": 8,
  "Name": "Memory DIMM Collection",
  "Oem": {
    "Hp": {
      "@odata.type": "#HpAdvancedMemoryProtection.1.0.0.HpAdvancedMemoryProtection",
      "AmpModeActive": "AdvancedECC",
      "AmpModeStatus": "AdvancedECC",
      "AmpModeSupported": [
        "AdvancedECC",
        "Lockstep",
        "OnlineSpareRank",
        "IntrasocketMirroring"
      ],
      "Type": "HpAdvancedMemoryProtection.1.0.0"
    }
  },
  "Total": 8,
  "Type": "Collection.1.0.0",
  "links": {
    "Member": [
      {
        "href": "/redfish/v1/Systems/1/Memory/board1dimm4/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board1dimm9/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board3dimm4/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board3dimm9/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board5dimm4/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board5dimm9/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board7dimm4/"
      },
      {
        "href": "/redfish/v1/Systems/1/Memory/board7dimm9/"
      }
    ],
    "self": {
      "href": "/redfish/v1/Systems/1/Memory/"
    }
  }
}

curl -k https://10.10.190.32/redfish/v1/Systems/1/Memory/board1dimm4/ |jq .

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/Memory/Members/$entity",
  "@odata.id": "/redfish/v1/Systems/1/Memory/board1dimm4/",
  "@odata.type": "#HpMemory.1.0.0.HpMemory",
  "DIMMStatus": "GoodInUse",
  "DIMMTechnology": "RDIMM",
  "DIMMType": "DDR4",
  "DataWidth": 64,
  "ErrorCorrection": "MultiBitECC",
  "HPMemoryType": "HPSmartMemory",
  "Id": "board1dimm4",
  "Manufacturer": "HP     ",
  "MaximumFrequencyMHz": 2400,
  "MinimumVoltageVoltsX10": 12,
  "Name": "board1dimm4",
  "PartNumber": "809082-091",
  "Rank": 1,
  "SizeMB": 16384,
  "SocketLocator": "BOARD 1 DIMM 4",
  "TotalWidth": 72,
  "Type": "HpMemory.1.0.0",
  "links": {
    "self": {
      "href": "/redfish/v1/Systems/1/Memory/board1dimm4/"
    }
  }
}

1

Disk:

curl -k https://10.10.190.32/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/ |jq .

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/SmartStorage/ArrayControllers/Members/$entity",
  "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/",
  "@odata.type": "#HpSmartStorageArrayController.1.1.0.HpSmartStorageArrayController",
  "AdapterType": "SmartArray",
  "BackupPowerSourceStatus": "NotPresent",
  "CurrentOperatingMode": "RAID",
  "Description": "HP Smart Storage Array Controller View",
  "EncryptionCryptoOfficerPasswordSet": false,
  "EncryptionEnabled": false,
  "EncryptionFwLocked": false,
  "EncryptionHasLockedVolumesMissingBootPassword": false,
  "EncryptionMixedVolumesEnabled": false,
  "EncryptionStandaloneModeEnabled": false,
  "ExternalPortCount": 0,
  "FirmwareVersion": {
    "Current": {
      "VersionString": "4.04"
    }
  },
  "HardwareRevision": "B",
  "Id": "0",
  "InternalPortCount": 2,
  "Location": "Slot 0",
  "LocationFormat": "PCISlot",
  "Model": "Smart Array P830i Controller",
  "Name": "HpSmartStorageArrayController",
  "SerialNumber": "001438040CF94C0",
  "Status": {
    "Health": "OK",
    "State": "Enabled"
  },
  "Type": "HpSmartStorageArrayController.1.1.0",
  "links": {
    "LogicalDrives": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/LogicalDrives/"
    },
    "PhysicalDrives": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/"
    },
    "StorageEnclosures": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/StorageEnclosures/"
    },
    "UnconfiguredDrives": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/UnconfiguredDrives/"
    },
    "self": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/"
    }
  }
}

curl -k https://10.10.190.32/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/ |jq .

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/SmartStorage/ArrayControllers/Members/0/DiskDrives",
  "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/",
  "@odata.type": "#HpSmartStorageDiskDriveCollection.HpSmartStorageDiskDriveCollection",
  "Description": "HP Smart Storage Disk Drives View",
  "MemberType": "HpSmartStorageDiskDrive.1",
  "Members": [
    {
      "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/0/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/1/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/2/"
    },
    {
      "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/3/"
    }
  ],
  "Members@odata.count": 4,
  "Name": "HpSmartStorageDiskDrives",
  "Total": 4,
  "Type": "Collection.1.0.0",
  "links": {
    "Member": [
      {
        "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/0/"
      },
      {
        "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/1/"
      },
      {
        "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/2/"
      },
      {
        "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/3/"
      }
    ],
    "self": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/"
    }
  }
}

curl -k https://10.10.190.32/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/0/ |jq .

{
  "@odata.context": "/redfish/v1/$metadata#Systems/Members/1/SmartStorage/ArrayControllers/Members/0/DiskDrives/Members/$entity",
  "@odata.id": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/0/",
  "@odata.type": "#HpSmartStorageDiskDrive.1.3.0.HpSmartStorageDiskDrive",
  "BlockSizeBytes": 512,
  "CapacityGB": 480,
  "CapacityLogicalBlocks": 937703088,
  "CapacityMiB": 457862,
  "CarrierApplicationVersion": "11",
  "CarrierAuthenticationStatus": "OK",
  "CurrentTemperatureCelsius": 28,
  "Description": "HP Smart Storage Disk Drive View",
  "DiskDriveStatusReasons": [
    "None"
  ],
  "EncryptedDrive": false,
  "FirmwareVersion": {
    "Current": {
      "VersionString": "HPG4"
    }
  },
  "Id": "0",
  "InterfaceSpeedMbps": 6000,
  "InterfaceType": "SATA",
  "Location": "1I:1:1",
  "LocationFormat": "ControllerPort:Box:Bay",
  "MaximumTemperatureCelsius": 46,
  "MediaType": "SSD",
  "Model": "VK000480GWEZC",
  "Name": "HpSmartStorageDiskDrive",
  "PowerOnHours": 52053,
  "SSDEnduranceUtilizationPercentage": 12,
  "SerialNumber": "173818EFDA2A",
  "Status": {
    "Health": "OK",
    "State": "Enabled"
  },
  "Type": "HpSmartStorageDiskDrive.1.3.0",
  "links": {
    "self": {
      "href": "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/0/"
    }
  }
}

1 2

mrlhansen commented 8 months ago

The old iLO 4 machines are also not fully compatible with the Redfish standard.

I made some changes, such that Memory partially works now, but for the network there is again not enough information available. I have not looked at the storage part yet.

ilanni2460 commented 8 months ago

The old iLO 4 machines are also not fully compatible with the Redfish standard.

I made some changes, such that Memory partially works now, but for the network there is again not enough information available. I have not looked at the storage part yet.

Already very good, I just used the latest code and can already get the latest memory information. Currently, it cannot be stored, so I don’t know if it can be obtained. It doesn't matter if the network cannot be obtained. After all, the network here only shows how many network ports are alive and the speed of the network card.

ilanni2460 commented 8 months ago

@mrlhansen There are some problems found in obtaining the memory operating frequency. In the ilo web interface, you can clearly see that the operating frequency is 2400MHz, but through idrac_exporter, the obtained operating frequency is 0. I guess this should be a bug.

At the same time, I also noticed that the value obtained by the metric idrac_power_control_capacity_watts is exactly the same as the memory operating frequency.

1 2

This is indeed a bug. I have fixed it. I need to modify line 364 of the internal/collector/client.go file and add the following content:

m.OperatingSpeedMhz = m.MaximumFrequencyMHz

1 2

mrlhansen commented 8 months ago

I was actually looking at this. However, 2400 Mhz is the maximal possible speed of the module, while in reality the actual speed is in your case only 1333 Mhz from what I can read. The metric is supposed to report the operating speed and not the maximal possible speed and this is why I did not include it.

ilanni2460 commented 8 months ago

I was actually looking at this. However, 2400 Mhz is the maximal possible speed of the module, while in reality the actual speed is in your case only 1333 Mhz from what I can read. The metric is supposed to report the operating speed and not the maximal possible speed and this is why I did not include it.

OK, thank you very much for your reply.

ilanni2460 commented 8 months ago

@mrlhansen By the way, I have updated part of the code for ilo4 storage monitoring. In line 131 of client.go, I added the following content. 

            client.memoryPath = "/redfish/v1/Systems/1/Memory/"
            client.storagePath = "/redfish/v1/Systems/1/SmartStorage/ArrayControllers/0/DiskDrives/"

1

After compilation, using the debug mode of idrac_exporter, you can see the interface for storing redfish, but I really don’t see the relevant information stored. Please help me find out where the problem is. Thank you very much🙏

2

mrlhansen commented 8 months ago

I made another commit and storage should now be working. I think this is the best I can do for iLO 4

ilanni2460 commented 8 months ago

I made another commit and storage should now be working. I think this is the best I can do for iLO 4

Thank you very much. Using the latest code, you can now obtain disk related information.

1