search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.24k stars 2.37k forks source link

delayed responces #228

Closed hryamzik closed 8 years ago

hryamzik commented 8 years ago

One of my node exporters stopped responding for the second time and I've done some investigation. In fact it does respond:

time curl -s 127.0.0.1:9100/metrics

Results in 1m48.837s This time is pretty stable. Here's some info from GDB:

Go routines:

  1 runnable runtime.gopark
* 17 syscall  runtime.goexit
  2 waiting  runtime.gopark
  3 waiting  runtime.gopark
  4 waiting  runtime.gopark
* 66 running  runtime.systemstack_switch
  67 waiting  runtime.gopark
  82 waiting  runtime.gopark
  98 waiting  runtime.gopark
  83 waiting  runtime.gopark
  84 waiting  runtime.gopark
  85 waiting  runtime.gopark
  86 waiting  runtime.gopark
  87 waiting  runtime.gopark
  88 waiting  runtime.gopark
  89 waiting  runtime.gopark
  90 waiting  runtime.gopark
  99 waiting  runtime.gopark
  91 waiting  runtime.gopark
  92 waiting  runtime.gopark
  93 waiting  runtime.gopark
  100 waiting  runtime.gopark
  210 waiting  runtime.gopark
  261733 waiting  runtime.gopark
  261732 runnable runtime.Gosched

261732 routine:

#0  runtime.Gosched () at /usr/local/go/src/runtime/proc.go:167
#1  0x000000000041a0b2 in runtime.startGC (mode=0, forceTrigger=false) at /usr/local/go/src/runtime/mgc.go:880
#2  0x0000000000411f7b in runtime.mallocgc (size=1048576, typ=0x8609a0, flags=1, ~r3=0xc8202cd680) at /usr/local/go/src/runtime/malloc.go:708
#3  0x0000000000412779 in runtime.newarray (typ=0x8609a0, n=131072, ~r2=0xc8202cd680) at /usr/local/go/src/runtime/malloc.go:780
#4  0x0000000000443305 in runtime.makeslice (t=0x84a000, len64=131072, cap64=131072, ~r3=...) at /usr/local/go/src/runtime/slice.go:32
#5  0x00000000007674f2 in compress/flate.(*compressor).init (d=0xc8205d8240, w=..., level=6, err=...) at /usr/local/go/src/compress/flate/deflate.go:410
#6  0x00000000007681e4 in compress/flate.NewWriter (w=..., level=-1, ~r2=0xa, ~r3=...) at /usr/local/go/src/compress/flate/deflate.go:487
#7  0x000000000060cb48 in compress/gzip.(*Writer).Write (z=0xc8200ce160, p= []uint8 = {...}, ~r1=2, ~r2=...) at /usr/local/go/src/compress/gzip/gzip.go:214
#8  0x00000000007c1543 in github.com/prometheus/node_exporter/vendor/github.com/matttproud/golang_protobuf_extensions/pbutil.WriteDelimited (w=..., m=..., n=4251990, err=...)
    at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/matttproud/golang_protobuf_extensions/pbutil/encode.go:39
#9  0x00000000006b0271 in github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt.NewEncoder.func1 (v=0xc820166910, ~r1=...) at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt/encode.go:68
#10 0x00000000006a40a2 in github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt.encoder.Encode (e={void (struct github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_model/go.MetricFamily *, error *)} 0xc820059358, 
    v=0xc820166910, ~r1=...) at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/expfmt/encode.go:36
#11 0x00000000004ea190 in github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB (r=0xc820014660, encoder=..., ~r1=...)
    at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:511
#12 0x00000000004e8983 in github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP (r=0xc820014660, w=..., req=0xc82037c000)
    at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:375
#13 0x00000000004f3d70 in net/http.(Handler).ServeHTTP-fm (a0=..., a1=0xc82037c000) at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:61
#14 0x00000000004f40b5 in github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1 (w=..., r=0xc82037c000)
    at /home/mr/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:158
#15 0x000000000049792a in net/http.HandlerFunc.ServeHTTP (f={void (net/http.ResponseWriter, struct net/http.Request *)} 0xc820059b78, w=..., r=0xc82037c000) at /usr/local/go/src/net/http/server.go:1422
#16 0x000000000049918d in net/http.(*ServeMux).ServeHTTP (mux=0xc82000a990, w=..., r=0xc82037c000) at /usr/local/go/src/net/http/server.go:1699
#17 0x0000000000499c4e in net/http.serverHandler.ServeHTTP (sh=..., rw=..., req=0xc82037c000) at /usr/local/go/src/net/http/server.go:1862
#18 0x00000000004973ee in net/http.(*conn).serve (c=0xc8200ce000) at /usr/local/go/src/net/http/server.go:1361
#19 0x0000000000460ed1 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1721
#20 0x000000c8200ce000 in ?? ()
#21 0x0000000000000000 in ?? ()

OS threads trace:

Thread 23 (Thread 0x7f7c0c597700 (LWP 101916)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b550 in runtime.futexsleep (addr=0xcba180 <runtime.sched+192>, val=0, ns=60000000000) at /usr/local/go/src/runtime/os1_linux.go:56
#2  0x0000000000410d6b in runtime.notetsleep_internal (n=0xcba180 <runtime.sched+192>, ns=60000000000, ~r2=false) at /usr/local/go/src/runtime/lock_futex.go:171
#3  0x0000000000410e5b in runtime.notetsleep (n=0xcba180 <runtime.sched+192>, ns=60000000000, ~r2=13) at /usr/local/go/src/runtime/lock_futex.go:191
#4  0x000000000043a1ba in runtime.sysmon () at /usr/local/go/src/runtime/proc1.go:3022
#5  0x0000000000433288 in runtime.mstart1 () at /usr/local/go/src/runtime/proc1.go:715
#6  0x0000000000433192 in runtime.mstart () at /usr/local/go/src/runtime/proc1.go:685
#7  0x0000000000825c24 in crosscall_amd64 ()
#8  0x00007f7c0c5979c0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7f7c0bd96700 (LWP 101917)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820038a10, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820038a10) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206ba4b0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206ba480) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820017500 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f7c0b595700 (LWP 101918)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820038e90, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820038e90) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043715c in runtime.exitsyscall0 (gp=0xc82030f800) at /usr/local/go/src/runtime/proc1.go:2146
#5  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#6  0x000000c820016000 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f7c0ad94700 (LWP 101919)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820148110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820148110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206ba630, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206ba600) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820018a00 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f7c0a553700 (LWP 101921)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820148a10, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820148a10) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206c0030, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206c0000) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82014b0c0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f7c09d52700 (LWP 101922)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820117310, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820117310) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043715c in runtime.exitsyscall0 (gp=0xc8204bbb00) at /usr/local/go/src/runtime/proc1.go:2146
#5  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#6  0x000000c82001a000 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f7c09511700 (LWP 101923)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b550 in runtime.futexsleep (addr=0xcba170 <runtime.sched+176>, val=0, ns=100000) at /usr/local/go/src/runtime/os1_linux.go:56
#2  0x0000000000410d6b in runtime.notetsleep_internal (n=0xcba170 <runtime.sched+176>, ns=100000, ~r2=false) at /usr/local/go/src/runtime/lock_futex.go:171
#3  0x0000000000410e5b in runtime.notetsleep (n=0xcba170 <runtime.sched+176>, ns=100000, ~r2=false) at /usr/local/go/src/runtime/lock_futex.go:191
#4  0x0000000000432d73 in runtime.stopTheWorldWithSema () at /usr/local/go/src/runtime/proc1.go:580
#5  0x000000000045e849 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:262
#6  0x0000000000433120 in runtime.startTheWorldWithSema () at /usr/local/go/src/runtime/proc1.go:603
#7  0x00007f7c095119c0 in ?? ()
#8  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f7c08d10700 (LWP 101924)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820248110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820248110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x0000000000434cc8 in runtime.gcstopm () at /usr/local/go/src/runtime/proc1.go:1329
#5  0x0000000000434ef5 in runtime.findrunnable (gp=0x1, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1390
#6  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#7  0x0000000000435e6b in runtime.park_m (gp=0xc820626c00) at /usr/local/go/src/runtime/proc1.go:1706
#8  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#9  0x000000c820018a00 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f7bf3fff700 (LWP 101925)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820256110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820256110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206c0030, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206c0000) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82001e000 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f7bf37fe700 (LWP 101926)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820248590, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820248590) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc82014a4b0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc82014a480) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820018a00 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f7bf2ffd700 (LWP 101928)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820256590, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820256590) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206ba030, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206ba000) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82001ca00 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f7bf27fc700 (LWP 101927)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8202ae110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8202ae110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc820306600) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82001f500 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f7bf1ffb700 (LWP 101930)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8202d4110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8202d4110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x430042 <runtime.printslice+98>, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc820000f00) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820023500 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f7bf17fa700 (LWP 101932)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc820117790, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc820117790) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043715c in runtime.exitsyscall0 (gp=0xc820345080) at /usr/local/go/src/runtime/proc1.go:2146
#5  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#6  0x000000c820016000 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f7bf0ff9700 (LWP 101933)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc82046e110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc82046e110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x0000000000434cc8 in runtime.gcstopm () at /usr/local/go/src/runtime/proc1.go:1329
#5  0x0000000000434ef5 in runtime.findrunnable (gp=0xc8206baab0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1390
#6  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#7  0x0000000000435e6b in runtime.park_m (gp=0xc8206baa80) at /usr/local/go/src/runtime/proc1.go:1706
#8  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#9  0x000000c820026000 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f7be3fff700 (LWP 101936)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8206ca110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8206ca110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043715c in runtime.exitsyscall0 (gp=0xc8204ba300) at /usr/local/go/src/runtime/proc1.go:2146
#5  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#6  0x000000c8206c04c0 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f7be37fe700 (LWP 101937)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8206dc110, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8206dc110) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc8202c6f00) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82002a000 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f7be2ffd700 (LWP 101951)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8206dc590, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8206dc590) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0xc8206ba1b0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000435e6b in runtime.park_m (gp=0xc8206ba180) at /usr/local/go/src/runtime/proc1.go:1706
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82002a000 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f7be27fc700 (LWP 101952)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8206ca590, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8206ca590) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x7f7c0cd57b78, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc8202e8900) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820017500 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f7be1ffb700 (LWP 101998)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc82046ea10, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc82046ea10) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc8201fdb00) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c82030e340 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f7be17fa700 (LWP 102033)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8202d5c10, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8202d5c10) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043715c in runtime.exitsyscall0 (gp=0xc82035a900) at /usr/local/go/src/runtime/proc1.go:2146
#5  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#6  0x000000c82001e000 in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f7be0ff9700 (LWP 105467)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xc8206dce90, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xc8206dce90) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc820557800) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x000000c820345e40 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f7c0cd92740 (LWP 101915)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:289
#1  0x000000000042b4b3 in runtime.futexsleep (addr=0xcbaab0 <runtime.m0+272>, val=0, ns=-1) at /usr/local/go/src/runtime/os1_linux.go:39
#2  0x0000000000410c04 in runtime.notesleep (n=0xcbaab0 <runtime.m0+272>) at /usr/local/go/src/runtime/lock_futex.go:142
#3  0x0000000000434272 in runtime.stopm () at /usr/local/go/src/runtime/proc1.go:1136
#4  0x000000000043555e in runtime.findrunnable (gp=0x0, inheritTime=false) at /usr/local/go/src/runtime/proc1.go:1538
#5  0x0000000000435be7 in runtime.schedule () at /usr/local/go/src/runtime/proc1.go:1647
#6  0x0000000000436242 in runtime.goexit0 (gp=0xc820229980) at /usr/local/go/src/runtime/proc1.go:1773
#7  0x000000000045e7ab in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:204
#8  0x0000000000cba440 in runtime.g0 ()
#9  0x00007fff269bd2a0 in ?? ()
#10 0x0000000000cba400 in ?? ()
#11 0x0000000000433192 in runtime.mstart () at /usr/local/go/src/runtime/proc1.go:685
#12 0x000000000045e698 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:120
#13 0x0000000000000003 in ?? ()
#14 0x00007fff269bd2e8 in ?? ()
#15 0x0000000000000003 in ?? ()
#16 0x00007fff269bd2e8 in ?? ()
#17 0x0000000000000000 in ?? ()

http.go158 is here

It seems to be a runtime issue. I'm going to build node exporter with 1.6.1 and give it a try but this issue could be difficult to reproduce.

juliusv commented 8 years ago

Thanks! Please try 1.6.2 instead of 1.6.1, because that has a bunch more issues fixed and just came out.

hryamzik commented 8 years ago

Well, now I have something similar on another server with node_exporter build with 1.6.2:

time curl 127.0.0.1:9100/metrics
curl: (56) Recv failure: Connection reset by peer

real    1m48.762s

Time is absolutely the same.

(gdb) thread apply all bt

Thread 1 (process 47084):
#0  runtime.futex () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sys_linux_amd64.s:307
#1  0x000000000042c473 in runtime.futexsleep (addr=0xd73828 <runtime.m0+264>, val=0, ns=-1) at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/os1_linux.go:40
#2  0x0000000000410cf4 in runtime.notesleep (n=0xd73828 <runtime.m0+264>) at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/lock_futex.go:145
#3  0x0000000000434a2b in runtime.stopm () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:1538
#4  0x0000000000435488 in runtime.gcstopm () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:1742
#5  0x000000000043628c in runtime.schedule () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:2039
#6  0x0000000000436834 in runtime.goschedImpl (gp=0xc820a41c80) at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:2155
#7  0x00000000004368d2 in runtime.gopreempt_m (gp=0xc820a41c80) at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:2170
#8  0x00000000004479cc in runtime.newstack () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/stack.go:932
#9  0x000000000045e81f in runtime.morestack () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/asm_amd64.s:359
#10 0x0000000000d73100 in runtime.work ()
#11 0x00007ffcfc0e9b00 in ?? ()
#12 0x0000000000d73180 in runtime.work ()
#13 0x0000000000433912 in runtime.mstart () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/proc.go:1068
#14 0x000000000045e568 in runtime.rt0_go () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/asm_amd64.s:149
#15 0x0000000000000003 in ?? ()
#16 0x00007ffcfc0e9b48 in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007ffcfc0e9b48 in ?? ()
#19 0x0000000000000000 in ?? ()
(gdb) info goroutines
  1 runnable runtime.createfing
  2 runnable runtime.gopark
  17 waiting  runtime.gopark
  18 waiting  runtime.gopark
  38 waiting  runtime.gopark
  82 waiting  runtime.gopark
  65 waiting  runtime.gopark
  81 waiting  runtime.gopark
  53261 runnable runtime.convT2E
  39 waiting  runtime.gopark
  40 waiting  runtime.gopark
  41 waiting  runtime.gopark
  42 waiting  runtime.gopark
  43 waiting  runtime.gopark
  44 waiting  runtime.gopark
  45 waiting  runtime.gopark
  46 waiting  runtime.gopark
  66 waiting  runtime.gopark
  47 waiting  runtime.gopark
  97 waiting  runtime.gopark
  98 waiting  runtime.gopark
  99 waiting  runtime.gopark
  100 waiting  runtime.gopark
  101 waiting  runtime.gopark
  102 waiting  runtime.gopark
  103 waiting  runtime.gopark
  67 waiting  runtime.gopark
  104 waiting  runtime.gopark
  83 waiting  runtime.gopark
  84 waiting  runtime.gopark
  85 waiting  runtime.gopark
  86 waiting  runtime.gopark
  87 waiting  runtime.gopark
  88 waiting  runtime.gopark
  89 waiting  runtime.gopark
  113 waiting  runtime.gopark
  114 waiting  runtime.gopark
  53289 runnable runtime.slicebytetostring
* 53282 running  runtime.systemstack_switch
  53283 runnable runtime.slicebytetostring
(gdb) goroutine 53282 bt
#0  runtime.systemstack_switch () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/asm_amd64.s:245
#1  0x000000000042f334 in runtime.startpanic () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:525
#2  0x000000000042f433 in runtime.throw (s="unexpected signal during runtime execution") at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:546
#3  0x0000000000443bca in runtime.sigpanic () at /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sigpanic_unix.go:12
#4  0x00000000000011fd in ?? ()
#5  0x0000000000000000 in ?? ()

And here's the panic log:

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: unexpected signal during runtime execution
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
runtime: unexpected return pc for sort.quickSort called from 0x0
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: unknown caller pc
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
fatal error: systemstack called from unexpected goroutine
fatal error: runtime·unlock: lock count
runtime: newstack called from g=0xc820991980
    m=0xc820549c00 m->curg=0x0 m->g0=0xc820580300 m->gsignal=0xc820580180
strings.Join(0xc820230ac8, 0x2, 0x2, 0xa61978, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strings/strings.go:363 +0xee
path.Join(0xc820230ac8, 0x2, 0x2, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/path/path.go:153 +0xfa
github.com/prometheus/node_exporter/collector.procFilePath(0xa69eb8, 0x4, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/paths.go:30 +0xd2
github.com/prometheus/node_exporter/collector.(*statCollector).Update(0xc820196d40, 0xc820471b60, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/stat_linux.go:89 +0x6f
main.execute(0x7ffcfc0e9f34, 0x4, 0x7f8157c87130, 0xc820196d40, 0xc820471b60)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc820471b60, 0xc820a06000, 0x7ffcfc0e9f34, 0x4, 0x7f8157c87130, 0xc820196d40)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137
fatal error: runtime: wrong goroutine in newstack
hryamzik commented 8 years ago

And one more panic from the initially reported server. This one crashed and was restarted by a supervisor.

fatal error: malloc deadlock

goroutine 340924 [running]:
runtime.throw(0xa7bc70, 0xf)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:547 +0x90 fp=0xc820292760 sp=0xc820292748
runtime.mallocgc(0x18, 0x9f77c0, 0x7f4100000000, 0xc820190180)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/malloc.go:544 +0x1cc fp=0xc820292838 sp=0xc820292760
runtime.newobject(0x9f77c0, 0x7f418af1a588)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/malloc.go:781 +0x42 fp=0xc820292860 sp=0xc820292838
runtime.convT2I(0x9f77c0, 0x9a9600, 0xd8fe48, 0xc8202928c8, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/iface.go:168 +0xf0 fp=0xc820292890 sp=0xc820292860
sort.Strings(0xd90008, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/sort/sort.go:305 +0x6c fp=0xc8202928e8 sp=0xc820292890
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.NewDesc(0xc8201900c0, 0x16, 0xc8207576e0, 0x2a, 0x0, 0x0, 0x0, 0x0, 0x2a)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/desc.go:139 +0xf3c fp=0xc820292ba0 sp=0xc8202928e8
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.NewUntyped(0xa68390, 0x4, 0xa6a9c8, 0x6, 0xc82022ffa0, 0xa, 0xc8207576e0, 0x2a, 0x0, 0x0, ...)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/untyped.go:51 +0xd1 fp=0xc820292c08 sp=0xc820292ba0
github.com/prometheus/node_exporter/collector.(*vmStatCollector).Update(0xd90008, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/vmstat_linux.go:59 +0x464 fp=0xc820292e00 sp=0xc820292c08
main.execute(0x7ffd10f57f55, 0x6, 0x7f418aede510, 0xd90008, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82 fp=0xc820292f50 sp=0xc820292e00
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f55, 0x6, 0x7f418aede510, 0xd90008)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49 fp=0xc820292f80 sp=0xc820292f50
runtime.goexit()
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820292f88 sp=0xc820292f80
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f418aedf710, 0x72, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201e1170, 0x72, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201e1170, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201e1110, 0x0, 0x7f418aedf808, 0xc82047a000)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201ac0e0, 0x456170, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc8201ac0e0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2427 +0x41
net/http.(*Server).Serve(0xc8201b0400, 0x7f418aedf7d0, 0xc8201ac0e0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServe(0xc8201b0400, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2098 +0x136
net/http.ListenAndServe(0xa61c38, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2195 +0x98
main.main()
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:168 +0x8fc

goroutine 341046 [runnable]:
regexp.(*bitState).reset(0xc820058af0, 0x8, 0x4)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/backtrack.go:96 +0x189
regexp.(*machine).backtrack(0xc82032e6c0, 0x7f418af1a620, 0xc82032e798, 0x0, 0x8, 0x4, 0x5)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/backtrack.go:324 +0x106
regexp.(*Regexp).doExecute(0xc8209cafa0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc82049d100, 0x8, 0x0, 0x4, ...)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/exec.go:449 +0x423
regexp.(*Regexp).replaceAll(0xc8209cafa0, 0x0, 0x0, 0x0, 0xc82049d100, 0x8, 0x4, 0xc8209599d0, 0x0, 0x0, ...)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/regexp.go:501 +0x12d
regexp.(*Regexp).ReplaceAllString(0xc8209cafa0, 0xc82049d100, 0x8, 0xa64ca0, 0x5, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/regexp.go:460 +0x133
github.com/prometheus/node_exporter/collector.parseMemInfo(0x7f418af1a000, 0xc8204b2018, 0xc8204b2018, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:105 +0x3ac
github.com/prometheus/node_exporter/collector.getMemInfo(0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:79 +0x116
github.com/prometheus/node_exporter/collector.(*meminfoCollector).Update(0xc8201ac060, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:52 +0x43
main.execute(0x7ffd10f57f14, 0x7, 0x7f418aede3a8, 0xc8201ac060, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f14, 0x7, 0x7f418aede3a8, 0xc8201ac060)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340981 [chan send]:
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc8201461c0, 0xc8201b46c0, 0x130)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:187 +0x1bb
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:146 +0x18d

goroutine 340984 [runnable]:
syscall.Syscall(0x0, 0x7, 0xc82056b000, 0x1000, 0x540, 0x1000, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x7, 0xc82056b000, 0x1000, 0x1000, 0x379, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x7, 0xc82056b000, 0x1000, 0x1000, 0xc82056a000, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc8201ac1c8, 0xc82056b000, 0x1000, 0x1000, 0x6401a7, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:228 +0x53
os.(*File).Read(0xc8201ac1c8, 0xc82056b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:95 +0x8a
bufio.(*Scanner).Scan(0xc820369900, 0x5)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:208 +0x97b
github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.FS.NewStat(0xa619a0, 0x5, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/stat.go:36 +0x1ee
github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.ProcStat.StartTime(0xd5d8, 0xc82049f8d0, 0xd, 0xc82049f8e0, 0x1, 0x1904, 0x1904, 0x1904, 0x0, 0xffffffffffffffff, ...)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/proc_stat.go:165 +0x49
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).processCollect(0xc820148480, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:127 +0x3fa
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).(github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.processCollect)-fm(0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:85 +0x2a
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).Collect(0xc820148480, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:103 +0x2a
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc82049e990, 0xc8201b4720, 0x7f418af1a810, 0xc820148480)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x58
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x34d

goroutine 340986 [semacquire]:
sync.runtime_Semacquire(0xc82055bcfc)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc82055bcf0)
    /usr/local/Cellar/go/1.6.2/libexec/src/sync/waitgroup.go:127 +0xb4
main.NodeCollector.Collect(0xc8201a2810, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:79 +0x163
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc82049e990, 0xc8201b4720, 0x7f418aede628, 0xc8201a2810)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x58
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x34d

goroutine 341047 [runnable]:
strings.Join(0xc8206efb38, 0x2, 0x2, 0xa61978, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strings/strings.go:369 +0x3c3
path.Join(0xc8206efb38, 0x2, 0x2, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/path/path.go:153 +0xfa
github.com/prometheus/node_exporter/collector.procFilePath(0xa68130, 0x7, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/paths.go:30 +0xd2
github.com/prometheus/node_exporter/collector.getNetDevStats(0xc8201b8dc0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netdev_linux.go:34 +0x51
github.com/prometheus/node_exporter/collector.(*netDevCollector).Update(0xc8201ab7e0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netdev_common.go:55 +0x56
main.execute(0x7ffd10f57f1c, 0x6, 0x7f418aede3d0, 0xc8201ab7e0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f1c, 0x6, 0x7f418aede3d0, 0xc8201ab7e0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341049 [runnable]:
github.com/prometheus/node_exporter/collector.parseMegaCliAdapter(0x7f418af1a000, 0xc8201ac1d8, 0xc8201ac1d8, 0x7f418af1a000, 0xc8201ac1d8)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:127
github.com/prometheus/node_exporter/collector.(*megaCliCollector).updateAdapter(0xc8201a3d70, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:171 +0x1a4
github.com/prometheus/node_exporter/collector.(*megaCliCollector).Update(0xc8201a3d70, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:74 +0x33
main.execute(0x7ffd10f57f64, 0x7, 0x7f418aede560, 0xc8201a3d70, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f64, 0x7, 0x7f418aede560, 0xc8201a3d70)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340980 [runnable]:
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB(0xc8200ac600, 0x7f418aedfb48, 0xc8203ba540, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:427 +0x407
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP(0xc8200ac600, 0x7f418aedfa60, 0xc8201ac020, 0xc8201461c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:375 +0x173
net/http.(Handler).ServeHTTP-fm(0x7f418aedfa60, 0xc8201ac020, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/h2_bundle.go:3847 +0x50
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:158 +0x336
net/http.HandlerFunc.ServeHTTP(0xc8201a4640, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc8200a25d0, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc8201b0400, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc8206a0080)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2137 +0x44e

goroutine 340922 [runnable]:
syscall.Syscall(0x0, 0x6, 0xc8203a2c87, 0x379, 0x379, 0x379, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x6, 0xc8203a2c87, 0x379, 0x379, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x6, 0xc8203a2c87, 0x379, 0x379, 0xb779e9, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc820032028, 0xc8203a2c87, 0x379, 0x379, 0x6401a7, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:228 +0x53
os.(*File).Read(0xc820032028, 0xc8203a2c87, 0x379, 0x379, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:95 +0x8a
bufio.(*Scanner).Scan(0xc820584ce8, 0xc8201b4720)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:208 +0x97b
github.com/prometheus/node_exporter/collector.(*statCollector).Update(0xc8201a8a40, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/stat_linux.go:96 +0x198
main.execute(0x7ffd10f57f34, 0x4, 0x7f418aede448, 0xc8201a8a40, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f34, 0x4, 0x7f418aede448, 0xc8201a8a40)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341051 [runnable]:
strconv.atof64exact(0x2b38, 0x0, 0x2b00, 0x0, 0x10000)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:369
strconv.atof64(0xc82066017c, 0x5, 0xc82061b4d0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:486 +0x106
strconv.ParseFloat(0xc82066017c, 0x5, 0x40, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:536 +0x82
github.com/prometheus/node_exporter/collector.(*diskstatsCollector).Update(0xc8201ab1c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/diskstats_linux.go:198 +0x70b
main.execute(0x7ffd10f57ee2, 0x9, 0x7f418aede2b8, 0xc8201ab1c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57ee2, 0x9, 0x7f418aede2b8, 0xc8201ab1c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340982 [semacquire]:
sync.runtime_Semacquire(0xc82049e99c)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc82049e990)
    /usr/local/Cellar/go/1.6.2/libexec/src/sync/waitgroup.go:127 +0xb4
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func1(0xc82049e990, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:409 +0x21
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:411 +0x271

goroutine 341052 [runnable]:
bufio.(*Scanner).Scan(0xc8209b0a08, 0xc820112d80)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:128
github.com/prometheus/node_exporter/collector.mountPointDetails(0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:87 +0x1e8
github.com/prometheus/node_exporter/collector.(*filesystemCollector).GetStats(0xc8201a88c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:40 +0x72
github.com/prometheus/node_exporter/collector.(*filesystemCollector).Update(0xc8201a88c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_common.go:109 +0x47
main.execute(0x7ffd10f57efb, 0xa, 0x7f418aede330, 0xc8201a88c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57efb, 0xa, 0x7f418aede330, 0xc8201a88c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341048 [runnable]:
strings.Join(0xc820295cb8, 0x2, 0x2, 0xa61978, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strings/strings.go:369 +0x3c3
path.Join(0xc820295cb8, 0x2, 0x2, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/path/path.go:153 +0xfa
github.com/prometheus/node_exporter/collector.sysFilePath(0xa76ae0, 0x9, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/paths.go:34 +0xd2
github.com/prometheus/node_exporter/collector.(*bondingCollector).Update(0xc8201a71b0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:61 +0x5c
main.execute(0x7ffd10f57f5c, 0x7, 0x7f418aede538, 0xc8201a71b0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f5c, 0x7, 0x7f418aede538, 0xc8201a71b0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341044 [runnable]:
runtime.SetFinalizer(0x983bc0, 0xc82009e000, 0x8f1c20, 0xb7a520)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/mfinal.go:368 +0x461
os.NewFile(0x9, 0xa6e770, 0xc, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:57 +0xfc
os.OpenFile(0xa6e770, 0xc, 0x0, 0x0, 0xc81ffb158f, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:123 +0x11f
os.Open(0xa6e770, 0xc, 0xc8206cbad0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:244 +0x48
io/ioutil.ReadDir(0xa6e770, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/io/ioutil/ioutil.go:101 +0x64
github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit.GetServices(0xa6e770, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit/runit.go:58 +0x90
github.com/prometheus/node_exporter/collector.(*runitCollector).Update(0xc8201dc2c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/runit.go:84 +0x67
main.execute(0x7ffd10f57f6c, 0x5, 0x7f418aede588, 0xc8201dc2c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f6c, 0x5, 0x7f418aede588, 0xc8201dc2c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340927 [runnable]:
github.com/prometheus/node_exporter/collector.parseNetStats(0x7f418af1a000, 0xc82009c4f8, 0xc8202e00a0, 0x11, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:117 +0x590
github.com/prometheus/node_exporter/collector.getNetStats(0xc8202e00a0, 0x11, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:96 +0x113
github.com/prometheus/node_exporter/collector.(*netStatCollector).Update(0xc8201ac070, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:50 +0x8a
main.execute(0x7ffd10f57f23, 0x7, 0x7f418aede3f8, 0xc8201ac070, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f23, 0x7, 0x7f418aede3f8, 0xc8201ac070)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341050 [runnable]:
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57ed8, 0x9, 0x7f418aede230, 0xc8201a6680)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:74
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137
brian-brazil commented 8 years ago

This sounds like a bug in Go, there was a similar bug fixed in 1.5.2. This should be reported upstream.

hryamzik commented 8 years ago

Are you talking about malloc deadlock or both?

brian-brazil commented 8 years ago

The malloc deadlock, the other one also smells like a Go bug.

hryamzik commented 8 years ago

Ok, created golang/go#15438 and golang/go#15436.

Yeah, run it under the race detector and double check all the unsafe and cgo usages. ©

I think I'll be able to try running with race detector tomorrow.

hryamzik commented 8 years ago

I've updated to latest version and do not face this any more. Unfortunately I don't have time for debugging this, closing.