Inspect current statistics on sockets in your system:
$ ss -s
Total: 1295
TCP: 71 (estab 20, closed 16, orphaned 0, timewait 2)
Transport Total IP IPv6
RAW 1 0 1
UDP 31 18 13
TCP 55 38 17
INET 87 56 31
FRAG 0 0 0
Generate some load and check statistics again e.g.:
$ go-wrk -c=4 -n=2000 http://localhost:9999/proxy?module=nodeNoVerify
...
$ ss -s
Total: 3329
TCP: 6283 (estab 2054, closed 4193, orphaned 0, timewait 4180)
Transport Total IP IPv6
RAW 1 0 1
UDP 31 18 13
TCP 2090 1054 1036
INET 2122 1072 1050
FRAG 0 0 0
Checkout lots of open connections between exporter_exporter and node_exporter:
$ ss -etnp | grep 9100
Checkout logs of exporter_exporter:
2020/01/15 12:15:01 http: Accept error: accept tcp [::]:9999: accept4: too many open files; retrying in 5ms
2020/01/15 12:15:01 http: proxy error: dial tcp: lookup localhost: device or resource busy
Expected behavior
One open connection to node_exporter
Actual behavior
Approx. 2000 open connections to node_exporter
Possible solution
This happens because http.Transport struct must be reused (as doc suggests) but new instance is created every time httpConfig.ServerHTTP method is invoked. Creating http.Transport with DisableKeepAlives: true also solves the issue
Description
exporter_exporter
leaks sockets when serving metrics for module withverify=false
.How to reproduce
Run
exporter_exporter
with some module configured like this:Inspect current statistics on sockets in your system:
Generate some load and check statistics again e.g.:
Checkout lots of open connections between
exporter_exporter
andnode_exporter
:Checkout logs of
exporter_exporter
:Expected behavior
One open connection to
node_exporter
Actual behavior
Approx. 2000 open connections to
node_exporter
Possible solution
This happens because
http.Transport
struct must be reused (as doc suggests) but new instance is created every timehttpConfig.ServerHTTP
method is invoked. Creatinghttp.Transport
withDisableKeepAlives: true
also solves the issue