owasp-amass / amass

In-depth attack surface mapping and asset discovery
https://owasp.org/www-project-amass/
Other
12.07k stars 1.88k forks source link

Feature Request: SOCKS5 proxy support #6

Open caffix opened 6 years ago

caffix commented 6 years ago

@superuser5 indicated that it would be great to have support for a socks proxy (like shadowsocks) or just to be able to specify which proxy to use; especially important when using active recon.

sea-erkin commented 6 years ago

Would it be enough to use the Go Proxy on the HTTP client's Transport struct? I was able to proxy traffic into Burp suite using this.

proxyUrl, err = url.Parse(proxy)
if err != nil {
    return errors.New("Could not parse proxy URL")
}

client := &http.Client{
    Timeout: 30 * time.Second,
    Transport: &http.Transport{
        DialContext:           DialContext,
        MaxIdleConns:          200,
        IdleConnTimeout:       90 * time.Second,
        TLSHandshakeTimeout:   10 * time.Second,
        ExpectContinueTimeout: 5 * time.Second,
        Proxy: http.ProxyURL(proxyUrl),
    },
}
riramar commented 6 years ago

Hi @sea-erkin where did you exactly use this code?

sea-erkin commented 6 years ago

@riramar https://github.com/OWASP/Amass/blob/master/amass/utils/network.go

I'm not sure if that's everywhere that makes http requests however but it at least covers a majority of the sources and I just hardcoded my proxy url string as a quick test

riramar commented 6 years ago

@sea-erkin Got the idea! Thanks!

emtunc commented 6 years ago

@sea-erkin @riramar could you share your code snip-it for getting this working with Burp please? I've spent an hour trying to get it to use Burp as a proxy but not having much luck - probably doing something silly!

Thank you.

sea-erkin commented 6 years ago

@emtunc Sure. If you look at https://github.com/OWASP/Amass/blob/master/amass/utils/network.go you can see a function GetWebPage(). When I tested I simply hardcoded my proxyUrl and it looked something similar to:

func GetWebPage(url string, hvals map[string]string) (string, error) {

    proxyUrl, err = url.Parse("localhost:8080")
    if err != nil {
            return "", errors.New("Could not parse proxy URL")
    }

    client := &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            DialContext:           DialContext,
            MaxIdleConns:          200,
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   10 * time.Second,
            ExpectContinueTimeout: 5 * time.Second,
            Proxy: http.ProxyURL(proxyUrl),
        },
    }
        // excluded code below
}
adamtimmins commented 5 years ago

@sea-erkin do you have an updated snippet for the latest version? Trying what you made but I'm getting errors.

sea-erkin commented 5 years ago

@adamtimmins You could try setting the proxy in the init function.

func init() {

    proxyUrl, _ := url.Parse("http://localhost:8080")

    jar, _ := cookiejar.New(nil)
    defaultClient = &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            DialContext: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
                DualStack: true,
            }).DialContext,
            MaxIdleConns:          200,
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   20 * time.Second,
            ExpectContinueTimeout: 20 * time.Second,
            TLSClientConfig:       &tls.Config{InsecureSkipVerify: true},
            Proxy:                 http.ProxyURL(proxyUrl),
        },
        Jar: jar,
    }
    defaultClient.Transport, _ = cfrt.New(defaultClient.Transport)
}
caffix commented 5 years ago

@sea-erkin @adamtimmins That will only cause the web activity to go through the proxy. Much more has to be done in order for all the traffic generated by Amass to be handled.

adamtimmins commented 5 years ago

@caffix can we expect an update with proxy implementation in the near future?

ngkogkos commented 5 years ago

I think a proxy only for web traffic for the time being will be more than enough as long as this is made clear by Amass to the user.

It's mainly the API services that appear to block IP addresses, and not the DNS resolvers, from my testing.

xuxiaoyuspallow commented 4 years ago

Excuse me, is the feature online now?

caffix commented 4 years ago

444 Could be of interest to this issue, yet SOCKS5 proxy support is still not implemented.

ip-rw commented 2 years ago

After a long rant about the lack of proxy support and how when I fixed it before it was all a bit of a mess, looking at it now everything is much tidier and the single HTTP client appears to use http.ProxyFromEnvironment so I guess nothing to see here :)

Set proxy using HTTP_PROXY HTTPS_PROXY environment vars.

caffix commented 2 years ago

@ip-rw are you on the Discord server? It would be great to have you join the development channel there

ip-rw commented 1 year ago

Amass is brilliant and yet I rarely use it - I spend my time looking tcpdump trying to work out what it's doing and why it's gone into single threaded mode...

I've not looked at the code in some time but the way you're traversing your way back up to ASN level and then liberally mapping your way back down (and stuff) is something that I think a lot of people do by hand, I know I do. Fierce (of rsnake fame but now quite different and maintained by somebody else) does something similar but I can't think of anything else that actually automates this in a satisfactory way.

I'd be very interested in adding a mode to use the information already gleaned to do flexible 'vhost' scanning with support for HTTP2/3. Among other things it's a very good way to find use for hostnames that perhaps no longer resolve, or go via a CDN/WAF. It's not a new concept but it's one that depends on exactly the sort of data you're collecting already. There really aren't many (public) tools that tackle this head on at the moment and there probably should be.

I LOVE netmap btw, I've a tonne of DNS data and a desire to make my own domaintools/shodan... Graph dbs are something that seem perfectly suited to storing these sorts of arbitrary, linked entities (I wonder if Cayley would enjoy a Clickhouse backend... I suspect it would in spite of things. It's so fast when it comes to reads some slow periodic updates wouldn't be the end of the world).

Sorry for the totally off topic comment. Discord is an abomination but I'll check it out.

I'll have a proper read thru current source (iirc you'd rewritten everything last I checked... you still use an EventBus pattern?) and see if I can do anything to remove the need for a global timeout option... as far as I could tell everything was going pretty well until we started checking SRV records/grabbing certificates when everything suddenly became single threaded and needed killing.

Super cool tool. Would love to help push it that little bit further - if not for the performance there wouldnt be other tools...