jenningsloy318 / redfish_exporter

exporter to get metrics from redfish based hardware such as lenovo/dell/superc servers
Apache License 2.0
70 stars 62 forks source link

collected metric collected before with the same name and label values #14

Closed eddyg closed 4 years ago

eddyg commented 4 years ago

I have been looking for a way to get metrics from HPE servers into Prometheus.

I had hoped that redfish_exporter might just work "out of the box", but unfortunately when I try to initiate a "scrape", it errors:

An error has occurred while serving metrics:

4 error(s) occurred:
* [from Gatherer #2] collected metric "redfish_chassis_power_powersupply_state" { label:<name:"chassis_id" value:"1" > label:<name:"power_supply" value:"HpServerPowerSupply" > label:<name:"power_supply_id" value:"" > label:<name:"resource" value:"power_supply" > gauge:<value:1 > } was collected before with the same name and label values
* [from Gatherer #2] collected metric "redfish_chassis_power_powersupply_health" { label:<name:"chassis_id" value:"1" > label:<name:"power_supply" value:"HpServerPowerSupply" > label:<name:"power_supply_id" value:"" > label:<name:"resource" value:"power_supply" > gauge:<value:1 > } was collected before with the same name and label values
* [from Gatherer #2] collected metric "redfish_chassis_power_powersupply_last_power_output_watts" { label:<name:"chassis_id" value:"1" > label:<name:"power_supply" value:"HpServerPowerSupply" > label:<name:"power_supply_id" value:"" > label:<name:"resource" value:"power_supply" > gauge:<value:258 > } was collected before with the same name and label values
* [from Gatherer #2] collected metric "redfish_chassis_power_powersupply_power_capacity_watts" { label:<name:"chassis_id" value:"1" > label:<name:"power_supply" value:"HpServerPowerSupply" > label:<name:"power_supply_id" value:"" > label:<name:"resource" value:"power_supply" > gauge:<value:460 > } was collected before with the same name and label values

Any idea on what it might take to make this work?

If it helps, when I manually query /redfish/v1/Chassis/1/Power/, this is what's returned (after being "pretty-printed" with jq so it's not just one long line):

{
  "@odata.context": "/redfish/v1/$metadata#Chassis/Members/1/Power$entity",
  "@odata.id": "/redfish/v1/Chassis/1/Power/",
  "@odata.type": "#Power.1.0.1.Power",
  "Id": "Power",
  "Name": "PowerMetrics",
  "Oem": {
    "Hp": {
      "@odata.type": "#HpPowerMetricsExt.1.2.0.HpPowerMetricsExt",
      "SNMPPowerThresholdAlert": {
        "DurationInMin": 0,
        "ThresholdWatts": 0,
        "Trigger": "Disabled"
      },
      "Type": "HpPowerMetricsExt.1.2.0",
      "links": {
        "FastPowerMeter": {
          "href": "/redfish/v1/Chassis/1/Power/FastPowerMeter/"
        },
        "FederatedGroupCapping": {
          "href": "/redfish/v1/Chassis/1/Power/FederatedGroupCapping/"
        },
        "PowerMeter": {
          "href": "/redfish/v1/Chassis/1/Power/PowerMeter/"
        }
      }
    }
  },
  "PowerCapacityWatts": 920,
  "PowerConsumedWatts": 268,
  "PowerControl": [
    {
      "PowerCapacityWatts": 920,
      "PowerConsumedWatts": 268,
      "PowerLimit": {
        "LimitInWatts": null
      },
      "PowerMetrics": {
        "AverageConsumedWatts": 265,
        "IntervalInMin": 20,
        "MaxConsumedWatts": 331,
        "MinConsumedWatts": 263
      }
    }
  ],
  "PowerLimit": {
    "LimitInWatts": null
  },
  "PowerMetrics": {
    "AverageConsumedWatts": 265,
    "IntervalInMin": 20,
    "MaxConsumedWatts": 331,
    "MinConsumedWatts": 263
  },
  "PowerSupplies": [
    {
      "FirmwareVersion": "1.00",
      "LastPowerOutputWatts": 263,
      "LineInputVoltage": 121,
      "LineInputVoltageType": "ACMidLine",
      "Model": "656362-B21",
      "Name": "HpServerPowerSupply",
      "Oem": {
        "Hp": {
          "@odata.type": "#HpServerPowerSupply.1.0.0.HpServerPowerSupply",
          "AveragePowerOutputWatts": 263,
          "BayNumber": 1,
          "HotplugCapable": true,
          "MaxPowerOutputWatts": 302,
          "Mismatched": false,
          "PowerSupplyStatus": {
            "State": "Ok"
          },
          "Type": "HpServerPowerSupply.1.0.0",
          "iPDU": {
            "Id": "1",
            "Model": "",
            "SerialNumber": "",
            "iPDUStatus": {
              "State": "Unknown"
            }
          },
          "iPDUCapable": true
        }
      },
      "PowerCapacityWatts": 460,
      "PowerSupplyType": "AC",
      "SerialNumber": "XXXXXXXXXXXXXX",
      "SparePartNumber": "660184-001",
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      }
    },
    {
      "FirmwareVersion": "1.00",
      "LastPowerOutputWatts": 5,
      "LineInputVoltage": 121,
      "LineInputVoltageType": "ACMidLine",
      "Model": "656362-B21",
      "Name": "HpServerPowerSupply",
      "Oem": {
        "Hp": {
          "@odata.type": "#HpServerPowerSupply.1.0.0.HpServerPowerSupply",
          "AveragePowerOutputWatts": 5,
          "BayNumber": 2,
          "HotplugCapable": true,
          "MaxPowerOutputWatts": 5,
          "Mismatched": false,
          "PowerSupplyStatus": {
            "State": "Ok"
          },
          "Type": "HpServerPowerSupply.1.0.0",
          "iPDU": {
            "Id": "2",
            "Model": "",
            "SerialNumber": "",
            "iPDUStatus": {
              "State": "Unknown"
            }
          },
          "iPDUCapable": true
        }
      },
      "PowerCapacityWatts": 460,
      "PowerSupplyType": "AC",
      "SerialNumber": "XXXXXXXXXXXXXX",
      "SparePartNumber": "660184-001",
      "Status": {
        "Health": "OK",
        "State": "Enabled"
      }
    }
  ],
  "Redundancy": [
    {
      "MaxNumSupported": 2,
      "MemberId": "0",
      "MinNumNeeded": 2,
      "Mode": "Failover",
      "Name": "PowerSupply Redundancy Group 1",
      "RedundancySet": [
        {
          "@odata.id": "/redfish/v1/Chassis/1/Power#/PowerSupplies/0"
        },
        {
          "@odata.id": "/redfish/v1/Chassis/1/Power#/PowerSupplies/1"
        }
      ]
    }
  ],
  "Type": "PowerMetrics.0.11.0",
  "links": {
    "self": {
      "href": "/redfish/v1/Chassis/1/Power/"
    }
  }
}
jenningsloy318 commented 4 years ago

HI @eddyg from what you pasted, I can see that the power supplies from HPE servers don't have ID items, but actually this implemetation will relay on this to distinguish different powersupplies, and the upstream this project depends on don't implemented the function to extract ID from OEM section. and maybe new version of firmware has conform to the spec of the redfish. if possible, please consider to upgrade your firmware firstly.

or you can commente out the powersupply code and re-compile it again

eddyg commented 4 years ago

Thanks for following up. I will check to see if there's any firmware updates available from HPE that might address this issue (since it sounds like this may be out-of-spec).

I did try commenting out the code around the call to chassis.Power() (in chassis_collector.go) and it was able to gather the other metrics, so at least that's something!

Thanks again.