search

mrlhansen / idrac_exporter

Simple iDRAC exporter for Prometheus
MIT License
129 stars 39 forks source link

iDRAC 7 Testing Results #3

Closed branttaylor closed 1 year ago

branttaylor commented 2 years ago

Hey @mrlhansen, cool project! I have an iDRAC 7 here in my home lab and thought I'd give this a shot since it also has Redfish (I enabled in the web UI). Out of the box, 2 of the 4 metrics "modules" work great: sensors and sel. The power metrics is just empty but returns no errors, while the system metrics module errors out with this:

# ./idrac_exporter -config ./idrac.yaml
2022/10/10 17:13:14 http: panic serving 127.0.0.1:61509: interface conversion: interface {} is nil, not string
goroutine 19 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1850 +0xb0
panic({0x102468960, 0x14000206720})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/panic.go:890 +0x258
main.redfishSystem(0x1400018a090)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/redfish.go:224 +0x594
main.collectMetrics({0x14000162284, 0xc})
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:35 +0x1fc
main.metricsHandler({0x1024b1510, 0x1400019e000}, 0x129df663b?)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:73 +0x68
net/http.HandlerFunc.ServeHTTP(0x14000149ad8?, {0x1024b1510?, 0x1400019e000?}, 0x0?)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2109 +0x38
net/http.(*ServeMux).ServeHTTP(0x1400016227d?, {0x1024b1510, 0x1400019e000}, 0x14000198000)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2487 +0x140
net/http.serverHandler.ServeHTTP({0x1400011f620?}, {0x1024b1510, 0x1400019e000}, 0x14000198000)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2947 +0x2c4
net/http.(*conn).serve(0x1400014ad20, {0x1024b1960, 0x1400011f530})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1991 +0x560
created by net/http.(*Server).Serve
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:3102 +0x444
2022/10/10 17:13:16 http: panic serving 127.0.0.1:61520: interface conversion: interface {} is nil, not string
goroutine 34 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1850 +0xb0
panic({0x102468960, 0x14000206f30})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/panic.go:890 +0x258
main.redfishSystem(0x1400018a090)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/redfish.go:224 +0x594
main.collectMetrics({0x14000222314, 0xc})
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:35 +0x1fc
main.metricsHandler({0x1024b1510, 0x140002320e0}, 0x129df663b?)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:73 +0x68
net/http.HandlerFunc.ServeHTTP(0x14000284ad8?, {0x1024b1510?, 0x140002320e0?}, 0x0?)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2109 +0x38
net/http.(*ServeMux).ServeHTTP(0x1400022230d?, {0x1024b1510, 0x140002320e0}, 0x14000224200)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2487 +0x140
net/http.serverHandler.ServeHTTP({0x140002067e0?}, {0x1024b1510, 0x140002320e0}, 0x14000224200)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2947 +0x2c4
net/http.(*conn).serve(0x1400022c0a0, {0x1024b1960, 0x1400011f530})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1991 +0x560
created by net/http.(*Server).Serve
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:3102 +0x444
2022/10/10 17:13:21 http: panic serving 127.0.0.1:61536: interface conversion: interface {} is nil, not string
goroutine 5 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1850 +0xb0
panic({0x102468960, 0x140001c4f00})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/runtime/panic.go:890 +0x258
main.redfishSystem(0x1400018a090)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/redfish.go:224 +0x594
main.collectMetrics({0x1400001a194, 0xc})
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:35 +0x1fc
main.metricsHandler({0x1024b1510, 0x1400007e0e0}, 0x129df663b?)
        /Users/brant.taylor/Documents/git-home/idrac_exporter/idrac_exporter.go:73 +0x68
net/http.HandlerFunc.ServeHTTP(0x14000062ad8?, {0x1024b1510?, 0x1400007e0e0?}, 0x0?)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2109 +0x38
net/http.(*ServeMux).ServeHTTP(0x1400001a18d?, {0x1024b1510, 0x1400007e0e0}, 0x1400030c000)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2487 +0x140
net/http.serverHandler.ServeHTTP({0x140000120f0?}, {0x1024b1510, 0x1400007e0e0}, 0x1400030c000)
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:2947 +0x2c4
net/http.(*conn).serve(0x14000308000, {0x1024b1960, 0x1400011f530})
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:1991 +0x560
created by net/http.(*Server).Serve
        /opt/homebrew/Cellar/go/1.19.1/libexec/src/net/http/server.go:3102 +0x444

I'm a bit of a Go noob so I just wanted to see if anything might stand out to you that could make power return anything and system start working. Nice work on the project!

mrlhansen commented 2 years ago

Hi @branttaylor

Sorry for the slow reply, on vacation this week.

I do not have access to any iDRAC 7 myself, so it has never been tested. From the output I can see where it fails, so probably the json output from iDRAC 7 has a slightly different format. I will try to see if I can find some specs for the iDRAC 7 redfish implementation. Otherwise, I can send you a few commands that you can use to pull the information I need.

mrlhansen commented 2 years ago

Hi @branttaylor

I am struggling a bit to find some detailed information for iDRAC 7. Could you try to send me the output from the following two commands.

curl -k https://<ip>/redfish/v1/Systems/System.Embedded.1 -u <username>
curl -k https://<ip>/redfish/v1/Chassis/System.Embedded.1/Power -u <username>

Replace <ip> with the IP address of iDRAC and <username> with the username used to login. You will be prompted for the login password. Hopefully this will be enough for me to go on.

branttaylor commented 2 years ago

No worries at all! Here is the output:

# curl -k https://<ip>/redfish/v1/Systems/System.Embedded.1 -u <username>

{
  "@odata.context": "/redfish/v1/$metadata#ComputerSystem.ComputerSystem",
  "@odata.id": "/redfish/v1/Systems/System.Embedded.1",
  "@odata.type": "#ComputerSystem.v1_5_0.ComputerSystem",
  "Actions": {
    "#ComputerSystem.Reset": {
      "ResetType@Redfish.AllowableValues": [
        "On",
        "ForceOff",
        "ForceRestart",
        "GracefulShutdown",
        "PushPowerButton",
        "Nmi"
      ],
      "target": "/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset"
    }
  },
  "AssetTag": "                              ",
  "Bios": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Bios"
  },
  "BiosVersion": "1.11.0",
  "Boot": {
    "BootOptions": {
      "@odata.id": "/redfish/v1/Systems/System.Embedded.1/BootOptions"
    },
    "BootOrder": [],
    "BootOrder@odata.count": 0,
    "BootSourceOverrideEnabled": "Once",
    "BootSourceOverrideTarget": "None",
    "BootSourceOverrideTarget@Redfish.AllowableValues": [
      "None",
      "Pxe",
      "Cd",
      "Floppy",
      "Hdd",
      "BiosSetup",
      "Utilities",
      "UefiTarget",
      "SDCard",
      "UefiHttp"
    ]
  },
  "Description": "Computer System which represents a machine (physical or virtual) and the local resources such as memory, cpu and other devices that can be accessed from that machine.",
  "EthernetInterfaces": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/EthernetInterfaces"
  },
  "HostName": "localhost",
  "HostWatchdogTimer": {
    "FunctionEnabled": false,
    "Status": {
      "State": "Disabled"
    },
    "TimeoutAction": "None"
  },
  "Id": "System.Embedded.1",
  "IndicatorLED": "Off",
  "Links": {
    "Chassis": [
      {
        "@odata.id": "/redfish/v1/Chassis/System.Embedded.1"
      }
    ],
    "Chassis@odata.count": 1,
    "CooledBy": [
      {
        "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Fans/0x17%7C%7CFan.Embedded.Sys%201"
      },
      {
        "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Fans/0x17%7C%7CFan.Embedded.Sys%202"
      },
      {
        "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Fans/0x17%7C%7CFan.Embedded.Sys%203"
      }
    ],
    "CooledBy@odata.count": 3,
    "ManagedBy": [
      {
        "@odata.id": "/redfish/v1/Managers/iDRAC.Embedded.1"
      }
    ],
    "ManagedBy@odata.count": 1,
    "Oem": {
      "DELL": {
        "@odata.type": "#DellComputerSystem.v1_0_0.DellComputerSystem",
        "BootOrder": {
          "@odata.id": "/redfish/v1/Systems/System.Embedded.1/BootSources"
        }
      }
    },
    "PoweredBy": [],
    "PoweredBy@odata.count": 0
  },
  "Manufacturer": "Dell Inc.",
  "Memory": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Memory"
  },
  "MemorySummary": {
    "Status": {
      "Health": "OK",
      "HealthRollup": "OK",
      "State": "Absent"
    },
    "TotalSystemMemoryGiB": 0
  },
  "Model": "PowerEdge R220",
  "Name": "System",
  "NetworkInterfaces": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/NetworkInterfaces"
  },
  "PCIeDevices": null,
  "PCIeDevices@odata.count": 0,
  "PCIeFunctions": null,
  "PCIeFunctions@odata.count": 0,
  "PartNumber": "081N4VA08",
  "PowerState": "On",
  "ProcessorSummary": {
    "Count": 0,
    "LogicalProcessorCount": 0,
    "Model": null,
    "Status": {
      "Health": "OK",
      "HealthRollup": "OK",
      "State": "Absent"
    }
  },
  "Processors": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Processors"
  },
  "SKU": "9QYZF42",
  "SerialNumber": "CN747515AT0288",
  "SimpleStorage": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/SimpleStorage/Controllers"
  },
  "Status": {
    "Health": "OK",
    "HealthRollup": "OK",
    "State": "Enabled"
  },
  "Storage": {
    "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Storage"
  },
  "SystemType": "Physical",
  "UUID": "4c4c4544-0051-5910-805a-b9c04f463432"
}
# curl -k https://<ip>/redfish/v1/Chassis/System.Embedded.1/Power -u <username>

{
  "@odata.context": "/redfish/v1/$metadata#Power.Power",
  "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Power",
  "@odata.type": "#Power.v1_5_0.Power",
  "Description": "Power",
  "Id": "Power",
  "Name": "Power",
  "PowerControl": [],
  "PowerControl@odata.count": 0,
  "PowerSupplies": [],
  "PowerSupplies@odata.count": 0,
  "Redundancy": [],
  "Redundancy@odata.count": 0,
  "Voltages": [
    {
      "@odata.context": "/redfish/v1/$metadata#Power.Power",
      "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Voltages/iDRAC.Embedded.1%23CPU1VCOREPG",
      "@odata.type": "#Power.v1_3_0.Voltage",
      "LowerThresholdCritical": null,
      "LowerThresholdFatal": null,
      "LowerThresholdNonCritical": null,
      "MemberId": "iDRAC.Embedded.1#CPU1VCOREPG",
      "Name": "CPU1 VCORE PG",
      "PhysicalContext": "CPU",
      "ReadingVolts": 1,
      "RelatedItem": [
        {
          "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Processors/CPU.Socket.1"
        }
      ],
      "RelatedItem@odata.count": 1,
      "SensorNumber": 35,
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      },
      "UpperThresholdCritical": null,
      "UpperThresholdFatal": null,
      "UpperThresholdNonCritical": null
    },
    {
      "@odata.context": "/redfish/v1/$metadata#Power.Power",
      "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Voltages/iDRAC.Embedded.1%231.05VPG",
      "@odata.type": "#Power.v1_3_0.Voltage",
      "LowerThresholdCritical": null,
      "LowerThresholdFatal": null,
      "LowerThresholdNonCritical": null,
      "MemberId": "iDRAC.Embedded.1#1.05VPG",
      "Name": "1.05V PG",
      "PhysicalContext": "Chassis",
      "ReadingVolts": 1,
      "RelatedItem": [
        {
          "@odata.id": "/redfish/v1/Chassis/System.Embedded.1"
        }
      ],
      "RelatedItem@odata.count": 1,
      "SensorNumber": 39,
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      },
      "UpperThresholdCritical": null,
      "UpperThresholdFatal": null,
      "UpperThresholdNonCritical": null
    },
    {
      "@odata.context": "/redfish/v1/$metadata#Power.Power",
      "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Voltages/iDRAC.Embedded.1%23CPU1MEMCPUVDDQPG",
      "@odata.type": "#Power.v1_3_0.Voltage",
      "LowerThresholdCritical": null,
      "LowerThresholdFatal": null,
      "LowerThresholdNonCritical": null,
      "MemberId": "iDRAC.Embedded.1#CPU1MEMCPUVDDQPG",
      "Name": "CPU1 MEM CPU VDDQ PG",
      "PhysicalContext": "CPU",
      "ReadingVolts": 1,
      "RelatedItem": [
        {
          "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Processors/CPU.Socket.1"
        }
      ],
      "RelatedItem@odata.count": 1,
      "SensorNumber": 30,
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      },
      "UpperThresholdCritical": null,
      "UpperThresholdFatal": null,
      "UpperThresholdNonCritical": null
    },
    {
      "@odata.context": "/redfish/v1/$metadata#Power.Power",
      "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Voltages/iDRAC.Embedded.1%23SystemBoard1.5VPG",
      "@odata.type": "#Power.v1_3_0.Voltage",
      "LowerThresholdCritical": null,
      "LowerThresholdFatal": null,
      "LowerThresholdNonCritical": null,
      "MemberId": "iDRAC.Embedded.1#SystemBoard1.5VPG",
      "Name": "System Board 1.5V PG",
      "PhysicalContext": "SystemBoard",
      "ReadingVolts": 1,
      "RelatedItem": [
        {
          "@odata.id": "/redfish/v1/Chassis/System.Embedded.1"
        }
      ],
      "RelatedItem@odata.count": 1,
      "SensorNumber": 40,
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      },
      "UpperThresholdCritical": null,
      "UpperThresholdFatal": null,
      "UpperThresholdNonCritical": null
    },
    {
      "@odata.context": "/redfish/v1/$metadata#Power.Power",
      "@odata.id": "/redfish/v1/Chassis/System.Embedded.1/Sensors/Voltages/iDRAC.Embedded.1%23CPU1MEMCPUVTTPG",
      "@odata.type": "#Power.v1_3_0.Voltage",
      "LowerThresholdCritical": null,
      "LowerThresholdFatal": null,
      "LowerThresholdNonCritical": null,
      "MemberId": "iDRAC.Embedded.1#CPU1MEMCPUVTTPG",
      "Name": "CPU1 MEM CPU VTT PG",
      "PhysicalContext": "CPU",
      "ReadingVolts": 1,
      "RelatedItem": [
        {
          "@odata.id": "/redfish/v1/Systems/System.Embedded.1/Processors/CPU.Socket.1"
        }
      ],
      "RelatedItem@odata.count": 1,
      "SensorNumber": 20,
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      },
      "UpperThresholdCritical": null,
      "UpperThresholdFatal": null,
      "UpperThresholdNonCritical": null
    }
  ],
  "Voltages@odata.count": 5
}

Thanks for the help!

mrlhansen commented 2 years ago

Hi @branttaylor

Thanks a lot!

I fixed the problem for the system metrics. However, I can see that several of the variables are simply reported as 0 or null in iDRAC 7 - perhaps the Redfish implementation is not complete for this version of iDRAC.

Regarding the power metrics. These are based on the following section from the JSON output of iDRAC:

...
  "PowerSupplies": [],
...

In your case nothing is reported about the PSUs, so I am afraid there isn't much I can do in this case.

branttaylor commented 1 year ago

Sorry for the delay, but your changes work great! Thanks for fixing that.