search

lovoo / ipmi_exporter

IPMI Exporter for prometheus.io, written in Go.
BSD 3-Clause "New" or "Revised" License
80 stars 32 forks source link

ipmi_exporter abnormal exit #38

Open dengxiafubi opened 5 years ago

dengxiafubi commented 5 years ago

Hi, @SqiSch Recently, If trigger an exception

  1. The output of "ipmitool sensor" on the node as follows: [root@node-17 opt]# ipmitool sensor SDRR successfully erased Err in cmd get sensor sdr info Get SDR 0000 command failed: Requested sensor, data, or record not found Get SDR 0000 command failed: Requested sensor, data, or record not found Get SDR 0000 command failed: Requested sensor, data, or record not found Get SDR 0000 command failed: Requested sensor, data, or record not found Get SDR 0000 command failed: Requested sensor, data, or record not found

  2. The ipmi_exporter may abnormal exit when I execute the command of "curl 127.0.0.1:9289/metrics" [root@node-17 opt]# ./ipmi_exporter_master INFO[0000] Starting IPMI Exporter (version=2.2.0, branch=master-new, revision=4c54ecc2bb53341cad990784aaba762e46b115fe) source=main.go:36 INFO[0000] Build context (go=go1.12.7, user=root@master, date=20190802-04:11:08) source=main.go:37 INFO[0000] Listening on :9289 /metrics source=main.go:57 panic: runtime error: index out of range

goroutine 12 [running]: github.com/lovoo/ipmi_exporter/collector.convertOutput(0xc0000ca9f0, 0x2, 0x2, 0xc0000ca9f0, 0x2, 0x2, 0x0, 0x0) /root/dev/go/src/github.com/lovoo/ipmi_exporter/collector/collector.go:77 +0x41b github.com/lovoo/ipmi_exporter/collector.(Exporter).Collect(0xc00000c4e0, 0xc0000aa1e0) /root/dev/go/src/github.com/lovoo/ipmi_exporter/collector/collector.go:159 +0x1cf github.com/lovoo/ipmi_exporter/vendor/github.com/prometheus/client_golang/prometheus.(Registry).Gather.func2(0xc000022260, 0xc0000aa1e0, 0x903180, 0xc00000c4e0) /root/dev/go/src/github.com/lovoo/ipmi_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61 created by github.com/lovoo/ipmi_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather /root/dev/go/src/github.com/lovoo/ipmi_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:380 +0x2e9

  1. I found the problem is array crossover via debug (1) Location of the calling function `(dlv) n

    github.com/lovoo/ipmi_exporter/collector.(*Exporter).Collect() /root/dev/go/src/github.com/lovoo/ipmi_exporter/collector/collector.go:156 (PC: 0x6dffcb) Warning: debugging optimized function 151: output, err := ipmiOutput(e.IPMIBinary + " sensor") 152: if err != nil { 153: log.Errorln(err) 154: } 155: splitted, err := splitOutput(output) => 156: if err != nil { 157: log.Errorln(err) 158: } 159: convertedOutput, err := convertOutput(splitted) 160: if err != nil { 161: log.Errorln(err) (2) Print the result of ipmitool sensor (dlv) p splitted [][]string len: 2, cap: 2, [ [ "SDRR successfully erased", ], [ "Err in cmd get sensor sdr info", ], ]`

(3) When the array splitted being passed into the funciton of convertOutput() `(dlv) n

github.com/lovoo/ipmi_exporter/collector.convertOutput() /root/dev/go/src/github.com/lovoo/ipmiexporter/collector/collector.go:74 (PC: 0x6dedd5) Warning: debugging optimized function 69: func convertOutput(result [][]string) (metrics []metric, err error) { 70: for , res := range result { 71: var value float64 72: var currentMetric metric 73: => 74: for n := range res { 75: res[n] = strings.TrimSpace(res[n]) 76: } 77: value, err = convertValue(res[1], res[2]) 78: if err != nil { 79: log.Errorf("could not parse ipmi output: %s", err) ` (4) The index 2 of res will cause "panic: runtime error: index out of range"

dengxiafubi commented 5 years ago

@SqiSch I fix it as above. If it is convenient for you, please give me some suggestions, thanks