Clistats not stopped on runner close

meme-lord commented 3 months ago

httpx version: v1.6.7

Current Behavior:

I'm trying to use httpx as a library with stats enabled. I want to close one runner and open another but I get this error:

panic: pattern "/metrics" (registered at /go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123) conflicts with pattern "/metrics" (registered at /go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123):
/metrics matches the same requests as /metrics

goroutine 1 [running]:
net/http.(*ServeMux).register(...)
    /usr/local/go/src/net/http/server.go:2738
net/http.HandleFunc({0x135588d?, 0x10?}, 0x271fcc0?)
    /usr/local/go/src/net/http/server.go:2732 +0x86
github.com/projectdiscovery/clistats.(*Statistics).Start(0xc00e668680)
    /go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123 +0x6f
github.com/projectdiscovery/httpx/runner.(*Runner).prepareInput(0xc006320000)
    /go/pkg/mod/github.com/projectdiscovery/httpx@v1.6.7/runner/runner.go:488 +0x5a2
github.com/projectdiscovery/httpx/runner.(*Runner).RunEnumeration(0xc006320000)
    /go/pkg/mod/github.com/projectdiscovery/httpx@v1.6.7/runner/runner.go:713 +0x48c

Expected Behavior:

No crash. The clistats server should be shutdown when .Close() is called on the runner. In runner.go the Close() function looks like:

func (r *Runner) Close() {
    // nolint:errcheck // ignore
    r.hm.Close()
    r.hp.Dialer.Close()
    r.ratelimiter.Stop()
    if r.options.HostMaxErrors >= 0 {
        r.HostErrorsCache.Purge()
    }
    if r.options.Screenshot {
        r.browser.Close()
    }
    if r.options.OnClose != nil {
        r.options.OnClose()
    }
}

It should include

if r.options.ShowStatistics {
    runner.stats.Stop()
}

Steps To Reproduce:

package main

import "github.com/projectdiscovery/httpx/runner"
func main(){
    options := runner.Options{
        Methods:                   "GET",
        Threads: 1,
        ShowStatistics:true,
        InputTargetHost: []string{"http://google.com"},
    }
    options.ValidateOptions()
    httpxRunner, _ := runner.New(&options)
    httpxRunner.RunEnumeration()
    httpxRunner.Close()
    httpxRunner, _ = runner.New(&options)
    httpxRunner.RunEnumeration()
}

Running this will crash with this output:

./lol
https://google.com
panic: pattern "/metrics" (registered at /home/chieftan/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123) conflicts with pattern "/metrics" (registered at /home/chieftan/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123):
/metrics matches the same requests as /metrics

goroutine 1 [running]:
net/http.(*ServeMux).register(...)
    /usr/lib/go/src/net/http/server.go:2738
net/http.HandleFunc({0x15c8709?, 0x10?}, 0xc000075008?)
    /usr/lib/go/src/net/http/server.go:2732 +0x86
github.com/projectdiscovery/clistats.(*Statistics).Start(0xc000863780)
    /home/user/go/pkg/mod/github.com/projectdiscovery/clistats@v0.0.20/clistats.go:123 +0x6f
github.com/projectdiscovery/httpx/runner.(*Runner).prepareInput(0xc0008b81a0)
    /home/user/go/pkg/mod/github.com/projectdiscovery/httpx@v1.6.7/runner/runner.go:488 +0x5a2
github.com/projectdiscovery/httpx/runner.(*Runner).RunEnumeration(0xc0008b81a0)
    /home/user/go/pkg/mod/github.com/projectdiscovery/httpx@v1.6.7/runner/runner.go:713 +0x48c
main.main()
    /tmp/lol/lol.go:16 +0x125

meme-lord commented 3 months ago

The apiServer in runner.httpApiEndpoint also needs to be shutdown if options.HttpApiEndpoint but runner/apiendpoint.go has no Close or Stop function right now

ehsandeep commented 3 months ago

This is now fixed in latest release - https://github.com/projectdiscovery/httpx/releases/tag/v1.6.8

projectdiscovery / httpx