I've been load testing node_exporter with vegeta:
austin@austin2:~$ time ./src/gopath/src/github.com/tsenart/vegeta/vegeta attack -targets=targets.txt -rate=1000 >/dev/null
On one machine running node_exporter-0.16.0 (compiled with go-1.10.1), I got a new crash:
time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 1.350612s: couldn't get SNTP reply: read udp 127.0.0.1:47369->127.0.0.1:123: i/o timeout" source="collector.go:132"
time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 2.034638s: couldn't get SNTP reply: read udp 127.0.0.1:52188->127.0.0.1:123: i/o timeout" source="collector.go:132"
time="2018-05-23T08:34:51Z" level=error msg="ERROR: ntp collector failed after 1.215859s: couldn't get SNTP reply: read udp 127.0.0.1:42102->127.0.0.1:123: i/o timeout" source="collector.go:132"
fatal error: schedule: holding locks
Originally reported at https://github.com/prometheus/node_exporter/issues/870#issuecomment-391485697
I've been load testing node_exporter with vegeta: austin@austin2:~$ time ./src/gopath/src/github.com/tsenart/vegeta/vegeta attack -targets=targets.txt -rate=1000 >/dev/null
On one machine running node_exporter-0.16.0 (compiled with go-1.10.1), I got a new crash: time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 1.350612s: couldn't get SNTP reply: read udp 127.0.0.1:47369->127.0.0.1:123: i/o timeout" source="collector.go:132" time="2018-05-23T08:29:48Z" level=error msg="ERROR: ntp collector failed after 2.034638s: couldn't get SNTP reply: read udp 127.0.0.1:52188->127.0.0.1:123: i/o timeout" source="collector.go:132" time="2018-05-23T08:34:51Z" level=error msg="ERROR: ntp collector failed after 1.215859s: couldn't get SNTP reply: read udp 127.0.0.1:42102->127.0.0.1:123: i/o timeout" source="collector.go:132" fatal error: schedule: holding locks
runtime stack: runtime.throw(0xab48ab, 0x17) /usr/lib/go/src/runtime/panic.go:616 +0x81 runtime.schedule() /usr/lib/go/src/runtime/proc.go:2489 +0x351 runtime.goexit0(0xc422972300) /usr/lib/go/src/runtime/proc.go:2713 +0x22f runtime.mcall(0x1000000010004b0) /usr/lib/go/src/runtime/asm_amd64.s:351 +0x5b
node_exporter options: command_args="--collector.textfile.directory=/var/lib/node_exporter/ --collector.bonding --collector.buddyinfo --collector.ntp"
Full stack is attached. node_exporter.log