Feature Request: SOCKS5 proxy support

[caffix]

@superuser5 indicated that it would be great to have support for a socks proxy (like shadowsocks) or just to be able to specify which proxy to use; especially important when using active recon.

[sea-erkin]

Would it be enough to use the Go Proxy on the HTTP client's Transport struct? I was able to proxy traffic into Burp suite using this.

proxyUrl, err = url.Parse(proxy)
if err != nil {
    return errors.New("Could not parse proxy URL")
}

client := &http.Client{
    Timeout: 30 * time.Second,
    Transport: &http.Transport{
        DialContext:           DialContext,
        MaxIdleConns:          200,
        IdleConnTimeout:       90 * time.Second,
        TLSHandshakeTimeout:   10 * time.Second,
        ExpectContinueTimeout: 5 * time.Second,
        Proxy: http.ProxyURL(proxyUrl),
    },
}

[riramar]

Hi @sea-erkin where did you exactly use this code?

[sea-erkin]

@riramar https://github.com/OWASP/Amass/blob/master/amass/utils/network.go

I'm not sure if that's everywhere that makes http requests however but it at least covers a majority of the sources and I just hardcoded my proxy url string as a quick test

[riramar]

@sea-erkin Got the idea! Thanks!

[emtunc]

@sea-erkin @riramar could you share your code snip-it for getting this working with Burp please? I've spent an hour trying to get it to use Burp as a proxy but not having much luck - probably doing something silly!

Thank you.

[sea-erkin]

@emtunc Sure. If you look at https://github.com/OWASP/Amass/blob/master/amass/utils/network.go you can see a function GetWebPage(). When I tested I simply hardcoded my proxyUrl and it looked something similar to:

func GetWebPage(url string, hvals map[string]string) (string, error) {

    proxyUrl, err = url.Parse("localhost:8080")
    if err != nil {
            return "", errors.New("Could not parse proxy URL")
    }

    client := &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            DialContext:           DialContext,
            MaxIdleConns:          200,
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   10 * time.Second,
            ExpectContinueTimeout: 5 * time.Second,
            Proxy: http.ProxyURL(proxyUrl),
        },
    }
        // excluded code below
}

[adamtimmins]

@sea-erkin do you have an updated snippet for the latest version? Trying what you made but I'm getting errors.

[sea-erkin]

@adamtimmins You could try setting the proxy in the init function.

func init() {

    proxyUrl, _ := url.Parse("http://localhost:8080")

    jar, _ := cookiejar.New(nil)
    defaultClient = &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            DialContext: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
                DualStack: true,
            }).DialContext,
            MaxIdleConns:          200,
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   20 * time.Second,
            ExpectContinueTimeout: 20 * time.Second,
            TLSClientConfig:       &tls.Config{InsecureSkipVerify: true},
            Proxy:                 http.ProxyURL(proxyUrl),
        },
        Jar: jar,
    }
    defaultClient.Transport, _ = cfrt.New(defaultClient.Transport)
}

[caffix]

@sea-erkin @adamtimmins That will only cause the web activity to go through the proxy. Much more has to be done in order for all the traffic generated by Amass to be handled.

[adamtimmins]

@caffix can we expect an update with proxy implementation in the near future?

[ngkogkos]

I think a proxy only for web traffic for the time being will be more than enough as long as this is made clear by Amass to the user.

It's mainly the API services that appear to block IP addresses, and not the DNS resolvers, from my testing.

[xuxiaoyuspallow]

Excuse me, is the feature online now?

[caffix]

444 Could be of interest to this issue, yet SOCKS5 proxy support is still not implemented.

[ip-rw]

After a long rant about the lack of proxy support and how when I fixed it before it was all a bit of a mess, looking at it now everything is much tidier and the single HTTP client appears to use http.ProxyFromEnvironment so I guess nothing to see here :)

Set proxy using HTTP_PROXY HTTPS_PROXY environment vars.

[caffix]

@ip-rw are you on the Discord server? It would be great to have you join the development channel there

[ip-rw]

Amass is brilliant and yet I rarely use it - I spend my time looking tcpdump trying to work out what it's doing and why it's gone into single threaded mode...

I've not looked at the code in some time but the way you're traversing your way back up to ASN level and then liberally mapping your way back down (and stuff) is something that I think a lot of people do by hand, I know I do. Fierce (of rsnake fame but now quite different and maintained by somebody else) does something similar but I can't think of anything else that actually automates this in a satisfactory way.

I'd be very interested in adding a mode to use the information already gleaned to do flexible 'vhost' scanning with support for HTTP2/3. Among other things it's a very good way to find use for hostnames that perhaps no longer resolve, or go via a CDN/WAF. It's not a new concept but it's one that depends on exactly the sort of data you're collecting already. There really aren't many (public) tools that tackle this head on at the moment and there probably should be.

I LOVE netmap btw, I've a tonne of DNS data and a desire to make my own domaintools/shodan... Graph dbs are something that seem perfectly suited to storing these sorts of arbitrary, linked entities (I wonder if Cayley would enjoy a Clickhouse backend... I suspect it would in spite of things. It's so fast when it comes to reads some slow periodic updates wouldn't be the end of the world).

Sorry for the totally off topic comment. Discord is an abomination but I'll check it out.

I'll have a proper read thru current source (iirc you'd rewritten everything last I checked... you still use an EventBus pattern?) and see if I can do anything to remove the need for a global timeout option... as far as I could tell everything was going pretty well until we started checking SRV records/grabbing certificates when everything suddenly became single threaded and needed killing.

Super cool tool. Would love to help push it that little bit further - if not for the performance there wouldnt be other tools...