projectdiscovery / nuclei

Nuclei is a fast, customizable vulnerability scanner powered by the global security community and built on a simple YAML-based DSL, enabling collaboration to tackle trending vulnerabilities on the internet. It helps you find vulnerabilities in your applications, APIs, networks, DNS, and cloud configurations.
https://docs.projectdiscovery.io/tools/nuclei
MIT License
20.6k stars 2.51k forks source link

Add optional flag to enable http connection reuse #2829

Open Ice3man543 opened 2 years ago

Ice3man543 commented 2 years ago

Please describe your feature request:

Recently, we removed default http connection pooling to reduce memory usage. An optional flag should be added to enable that behaviour.

Describe the use case of this feature:

Mzack9999 commented 2 years ago

Each template process targets sequentially, so connections to a target remain open in a large number before being reused in another template for the same target. This boils down to a lot of memory usage and so considerable delay between connections to the same target that they become idle/broken. To exploit better connection reuse, we should change the approach and iterate templates over a target rather than iterating targets over a template. In this way, connections would be subsequent and reused without becoming idle/broken.

vzamanillo commented 1 year ago

Having MaxIdleConnsPerHost < 0 already prevents connection pooling

https://github.com/golang/go/blob/fbf763fd1d6be3c162ea5ff3c8843171ef937c3a/src/net/http/transport.go#L917

I am not sure if the recent change makes any difference, let me know if I am wrong

    connectionConfiguration := &httpclientpool.Configuration{
        Connection:   &httpclientpool.ConnectionConfiguration{DisableKeepAlive: true},
    }

Also having connection pooling enabled is good in terms of performance, maybe clearing the connections in the idle pool

    tr := &http.Transport{
        IdleConnTimeout:     10 * time.Second, (defaults to 90)
    }

or manually clearing the idle connections per template

request.httpClient.HTTPClient.CloseIdleConnections()

would work with a large URL set.