QubitProducts / exporter_exporter

A reverse proxy designed for Prometheus exporters
Apache License 2.0
336 stars 55 forks source link

sockets leak when fetching module with verify=false #37

Closed cyril-s closed 4 years ago

cyril-s commented 4 years ago

Description

exporter_exporter leaks sockets when serving metrics for module with verify=false.

How to reproduce

Run exporter_exporter with some module configured like this:

modules:
  nodeNoVerify:
    method: http
    http:
      verify: false
      port: 9100

Inspect current statistics on sockets in your system:

$ ss -s
Total: 1295
TCP:   71 (estab 20, closed 16, orphaned 0, timewait 2)

Transport Total     IP        IPv6
RAW   1         0         1        
UDP   31        18        13       
TCP   55        38        17       
INET      87        56        31       
FRAG      0         0         0        

Generate some load and check statistics again e.g.:

$ go-wrk -c=4 -n=2000 http://localhost:9999/proxy?module=nodeNoVerify
...
$ ss -s
Total: 3329
TCP:   6283 (estab 2054, closed 4193, orphaned 0, timewait 4180)

Transport Total     IP        IPv6
RAW   1         0         1        
UDP   31        18        13       
TCP   2090      1054      1036     
INET      2122      1072      1050     
FRAG      0         0         0  

Checkout lots of open connections between exporter_exporter and node_exporter:

$ ss -etnp | grep 9100

Checkout logs of exporter_exporter:

2020/01/15 12:15:01 http: Accept error: accept tcp [::]:9999: accept4: too many open files; retrying in 5ms
2020/01/15 12:15:01 http: proxy error: dial tcp: lookup localhost: device or resource busy

Expected behavior

One open connection to node_exporter

Actual behavior

Approx. 2000 open connections to node_exporter

Possible solution

This happens because http.Transport struct must be reused (as doc suggests) but new instance is created every time httpConfig.ServerHTTP method is invoked. Creating http.Transport with DisableKeepAlives: true also solves the issue

tcolgate commented 4 years ago

good catch, feel free to raise a PR, failing that, I'll take a look at the end of the week.