gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.2k stars 1.76k forks source link

How to set up only one request to use the proxy? #56

Closed Wangzhijie2016 closed 6 years ago

Wangzhijie2016 commented 6 years ago

I found

rp, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:1337", "socks5://127.0.0.1:1338")
    if err != nil {
        log.Fatal(err)
    }
c.SetProxyFunc(rp)

but if i have ten urls need request, this way is all request to use proxy, i only want one request to use the proxy, What should I do?

asciimoo commented 6 years ago

Ohi, Proxying only one request can be achieved multiple ways. You can use two collectors, one with proxy and another without proxy:

c := colly.NewCollector()
c2 := colly.NewCollector()
c2.SetProxy("http://127.0.0.1:8080/")
// ...

or you can unset the proxy after the first request:

c := colly.NewCollector()
c.SetProxy("http://127.0.0.1:8080")
c.Visit("http://example.com/")
c.SetProxyFunc(nil)
// ...
Wangzhijie2016 commented 6 years ago

Ok i try it, thanks