search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.69k stars 5.59k forks source link

Plugin: ipmi_sensor: fails to detect psu failure #5755

Closed vvershkov closed 5 years ago

vvershkov commented 5 years ago

Relevant telegraf.conf:

interval = "60s"
metric_version = 1
timeout = "10s"

System info:

Any OS or telegraf version: this bug is caused due to ipmi itself

Steps to reproduce:

We have a server with failed PSU: we know it because we saw it: it has amber light instead of green and sound alarm too. But ipmi and telegraf detect it status as OK:

...
PS1 Status       | 0x03              | ok
PS2 Status       | 0x01              | ok

However 0x03 flag is "failure": https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-sg8039en_us&docLocale=en_US

Expected behavior:

PS1 Status is bad (0)

Actual behavior:

PS1 Status is OK (1)

Additional info:

AFAIK there is no readings with 0x03 and "OK" status. for old motherboards - 0x03 for CPU for example means overheating. But maybe flag check needed only for PSU.

danielnelson commented 5 years ago

It looks like we don't handle hexadecimal values currently. Could you also run this command and add the output for these sensors so I can be sure to fix it for metric_version = 2 as well:

ipmitool sdr elist
vvershkov commented 5 years ago

Wow, that is a worst timing ever :) I just replaced it with new ones. But I can try to reproduce this bug with dead PSU and some server in office tomorrow.

vvershkov commented 5 years ago

Ok, ipmitool sdr and sdr elist outputs looks like this: Old version of ipmi failed PSU:

PS Status        | 0x02              | ok
PS Status        | 55h | ok  | 10.1 | Failure detected

(both PSU in one status, this is horrible)

after bmc update, turning off one power line:

PS1 Status       | 0x0b              | ok
PS2 Status       | 0x01              | ok
PS1 Status       | C8h | ok  | 10.1 | Presence detected, Failure detected, Power Supply AC lost
PS2 Status       | C9h | ok  | 10.2 | Presence detected

switching to failed psu:

PS1 Status       | 0x03              | ok
PS2 Status       | 0x01              | ok
PS1 Status       | C8h | ok  | 10.1 | Presence detected, Failure detected
PS2 Status       | C9h | ok  | 10.2 | Presence detected

PS funny thing:

PS1 Status       | 0x00              | ok
PS2 Status       | 0x01              | ok
PS1 Status       | C8h | ok  | 10.1 | 
PS2 Status       | C9h | ok  | 10.2 | Presence detected

PS1 is missing but its OK :)

danielnelson commented 5 years ago

@vvershkov We are finalizing a fix for the value, but you should know it will still report these examples as status=1. This is because we don't want to attempt to do any special interpretation, I don't think this would be feasible for all producets, and will just pass the info along.

You will need to setup your alerts to handle this specific issue either with the value in metric_version = 1 or with the status_desc with metric_version = 2.