Tune http.Client's Transport's MaxIdleConns/MaxIdleConnsPerHost

0xdevalias commented 5 years ago

In doing some work on another project, i've learnt far more than I ever expected to about go's http.Client, and how it's defaults may not be ideal. We're already setting timeouts, which is good, but the defaults around idle connection reuse are pretty woeful:

This is our current settings:

https://github.com/OJ/gobuster/blob/master/libgobuster/http.go#L51-L59

Some refs:

Looking at the source for net/http/transport.go we can see that the MaxIdleConnections is 100, but DefaultMaxIdleConnsPerHost is only 2:

var DefaultTransport RoundTripper = &Transport{
    Proxy: ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
        DualStack: true,
    }).DialContext,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}

// DefaultMaxIdleConnsPerHost is the default value of Transport's
// MaxIdleConnsPerHost.
const DefaultMaxIdleConnsPerHost = 2

What this means in practical terms is that for every 100 connections we open to a webserver, we close 98 of them, which leads to a lot of sockets left in our kernel in the TIME_WAITstate, and also means we need to pay the overhead of establishing a new TCP (and/or TLS) handshake on those 98 closed sockets.

I'm not 100% on this, but I think the best setting would be to set MaxIdleConns == MaxIdleConnsPerHost == number of goroutines/'threads' specified.

https://github.com/golang/go/blob/master/src/net/http/transport.go#L191-L216

Then if we make sure that we set keep-alive on requests to the server, if supported, it should mean that we get a lot more reuse per socket, which should mean faster gobusting.

For this to work, you also have to ensure you fully read/close the response body:

Just be sure to fully read (and/or close) the Response.Body, otherwise that TCP connection won't be re-used.

It looks as though the entire body is already being read, in these areas, so this should be fine:

There are a LOT more little things that can be tuned in the http.Client's Transport, but not sure if/what would make as much difference as this.

Making use of keep-alive was one of the techniques discussed in this research:

If anyone wants to benchmark the speed/resource difference before/after, i'd be pretty interested to see just how much of an improvement it makes, but from the above video, it claims about a 400% increase from reusing the connections (and like 6000% if you want to use http pipelining..)

OJ commented 5 years ago

I'll try to get to this very soon. Nice research @0xdevalias thank you!

iSteed commented 5 years ago

came here to ask about this, i am looking for a simple way to control the amount of connections i am establishing via the command line.

firefart commented 5 years ago

Did some quick testing on windows and there is indeed a speedup.

Default values from the optimization branch:

PS C:\Users\firefart\Documents\code\gobuster> Measure-Command { .\v3.exe dir -u https://firefart.at -w .\wordlist_small.txt }
2019/04/17 21:12:23 Starting gobuster
2019/04/17 21:12:39 Finished%)

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 16
Milliseconds      : 702
Ticks             : 167029543
TotalDays         : 0,000193321230324074
TotalHours        : 0,00463970952777778
TotalMinutes      : 0,278382571666667
TotalSeconds      : 16,7029543
TotalMilliseconds : 16702,9543

PS C:\Users\firefart\Documents\code\gobuster> Measure-Command { .\v3.exe dir -u https://firefart.at -w .\wordlist_small.txt }
2019/04/17 21:12:42 Starting gobuster
2019/04/17 21:12:59 Finished%)

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 16
Milliseconds      : 918
Ticks             : 169185885
TotalDays         : 0,000195816996527778
TotalHours        : 0,00469960791666667
TotalMinutes      : 0,281976475
TotalSeconds      : 16,9185885
TotalMilliseconds : 16918,5885

And after setting the following values:

    client.client = &http.Client{
        Timeout:       opt.Timeout,
        CheckRedirect: redirectFunc,
        Transport: &http.Transport{
            Proxy:               proxyURLFunc,
            MaxIdleConns:        100,
            MaxIdleConnsPerHost: 100,
            TLSClientConfig: &tls.Config{
                InsecureSkipVerify: opt.InsecureSSL,
            },
        }}

PS C:\Users\firefart\Documents\code\gobuster> Measure-Command { .\v3.exe dir -u https://firefart.at -w .\wordlist_small.txt }
2019/04/17 21:13:12 Starting gobuster
2019/04/17 21:13:27 Finished%)

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 14
Milliseconds      : 976
Ticks             : 149768399
TotalDays         : 0,000173343054398148
TotalHours        : 0,00416023330555556
TotalMinutes      : 0,249613998333333
TotalSeconds      : 14,9768399
TotalMilliseconds : 14976,8399

PS C:\Users\firefart\Documents\code\gobuster> Measure-Command { .\v3.exe dir -u https://firefart.at -w .\wordlist_small.txt }
2019/04/17 21:13:29 Starting gobuster
2019/04/17 21:13:44 Finished%)

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 14
Milliseconds      : 804
Ticks             : 148046087
TotalDays         : 0,000171349637731481
TotalHours        : 0,00411239130555556
TotalMinutes      : 0,246743478333333
TotalSeconds      : 14,8046087
TotalMilliseconds : 14804,6087

So with default parameters it takes around 17 seconds for a 5000 entry wordlist and with both parameters set to 100 we are around 15 seconds.

So I think we should use this optimization and hope we will not kill many servers with it :)

firefart commented 5 years ago

I created a PR over here: https://github.com/OJ/gobuster/pull/140

OJ / gobuster

Tune http.Client's Transport's MaxIdleConns/MaxIdleConnsPerHost #127