search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.14k stars 2.35k forks source link

fatal error: schedule: holding locks #955

Closed austin987 closed 5 years ago

austin987 commented 6 years ago

Originally reported at https://github.com/prometheus/node_exporter/issues/870#issuecomment-391485697

I've been load testing node_exporter with vegeta: austin@austin2:~$ time ./src/gopath/src/github.com/tsenart/vegeta/vegeta attack -targets=targets.txt -rate=1000 >/dev/null

On one machine running node_exporter-0.16.0 (compiled with go-1.10.1), I got a new crash: time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 1.350612s: couldn't get SNTP reply: read udp 127.0.0.1:47369->127.0.0.1:123: i/o timeout" source="collector.go:132" time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 2.034638s: couldn't get SNTP reply: read udp 127.0.0.1:52188->127.0.0.1:123: i/o timeout" source="collector.go:132" time="2018-05-23T08:34:51Z" level=error msg="ERROR: ntp collector failed after 1.215859s: couldn't get SNTP reply: read udp 127.0.0.1:42102->127.0.0.1:123: i/o timeout" source="collector.go:132" fatal error: schedule: holding locks

runtime stack: runtime.throw(0xab48ab, 0x17) /usr/lib/go/src/runtime/panic.go:616 +0x81 runtime.schedule() /usr/lib/go/src/runtime/proc.go:2489 +0x351 runtime.goexit0(0xc422972300) /usr/lib/go/src/runtime/proc.go:2713 +0x22f runtime.mcall(0x1000000010004b0) /usr/lib/go/src/runtime/asm_amd64.s:351 +0x5b

node_exporter options: command_args="--collector.textfile.directory=/var/lib/node_exporter/ --collector.bonding --collector.buddyinfo --collector.ntp"

Full stack is attached. node_exporter.log

discordianfish commented 5 years ago

I think this should be fixed by the fixes for the wifi collector.