camerondavison
commented
9 years ago After restarting it everything looks rather normal again. I tried to do a kill -QUIT and get a stack trace, but the trace itself was printing incredibly slowly and then I think that my systemd definitions killed it and restarted it in the middle. So I did not get anything.
Before I restarted it I ran strace -f -p $PID and it was only printing
Process 25359 attached with 7 threads
[pid 5361] futex(0xc2085680d8, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 23093] futex(0xc20837c458, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25364] futex(0xc208021258, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25363] futex(0xd18520, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25359] futex(0xd0f378, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631679, 472159297}) = 0
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631679, 472233554}) = 0
[pid 25362] clock_gettime(CLOCK_REALTIME, {1431543852, 511458926}) = 0
[pid 25362] epoll_wait(5, {}, 128, 0) = 0
[pid 25362] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631680, 423690670}) = 0
[pid 25362] futex(0xd0e7a0, FUTEX_WAIT, 0, {60, 0}) = -1 ETIMEDOUT (Connection timed out)
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631741, 549970095}) = 0
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631741, 816163981}) = 0
[pid 25362] clock_gettime(CLOCK_REALTIME, {1431543914, 227795643}) = 0
[pid 25362] epoll_wait(5, {}, 128, 0) = 0
[pid 25362] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631742, 140082541}) = 0
[pid 25362] futex(0xd0e7a0, FUTEX_WAIT, 0, {60, 0}) = -1 ETIMEDOUT (Connection timed out)
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631803, 334995979}) = 0
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631803, 335140475}) = 0
[pid 25362] clock_gettime(CLOCK_REALTIME, {1431543975, 609309540}) = 0
[pid 25362] epoll_wait(5, {}, 128, 0) = 0
[pid 25362] select(0, NULL, NULL, NULL, {0, 20}) = 0 (Timeout)
[pid 25362] clock_gettime(CLOCK_MONOTONIC, {1631803, 521924470}) = 0
[pid 25362] futex(0xd0e7a0, FUTEX_WAIT, 0, {60, 0}Once I restarted it it was printing the expected open/read of the file descriptors every 5 seconds when my prometheus app polled it for data
juliusv
beorn7
I have been running the node_exporter version 0.8.0 for a while now. Maybe 2 weeks, and I am now seeing a pretty high IO load. If I run iotop then I am seeing it reading megabytes of data in a matter of seconds. It is basically using 99% of the io on the system.
If I run lsof -p $PID then I see that there are a lot of open file descriptors to /pro/stat and /proc/$PID/limits and /proc/meminfo and /proc/$PID/stat
if this helps
sudo lsof -p 25359 | awk '{print $9}' | sort | uniq -c | sort -n