search

cardigann / go-cloudflare-scraper

A golang http.Transport layer that uses Otto to solve Cloudflare challenges
130 stars 77 forks source link

Proxy #2

Open iMaxopoly opened 7 years ago

iMaxopoly commented 7 years ago

Is it by design to pass the proxy like so?:

Transport: scraper.NewTransport(&http.Transport{Proxy: http.ProxyURL(proxy)}),

Curious to know what your reply is because I tend to get goroutine crashes when I run multiple.

Thanks

paulm17 commented 6 years ago

I could not work out how to set a proxy with RoundTripper.

However, as a workaround I'm doing:

os.Setenv("HTTP_PROXY", "http://"+host)

Would defintely welcome code or an update to have this set natively via the transport.

Finally, I am using this multiple and I have no issues with goruntine crashes.