dmitshur
commented
4 years ago Is this issue possible to reproduce? It would be helpful to provide a snippet that can reproduce it, then it's easier to investigate this issue.
Open vlourme opened 4 years ago
dmitshur
commented
4 years ago Is this issue possible to reproduce? It would be helpful to provide a snippet that can reproduce it, then it's easier to investigate this issue.
vlourme
commented
4 years ago Here is a sample of how I use proxies:
client := http.Client{
Transport: &http.Transport{
Proxy: func(request *http.Request) (url *url.URL, err error) {
// Wrong proxy
return url.Parse("http://103.69.216.125:8080")
},
},
Timeout: 10 * time.Second,
}
resp, err := client.Get("https://httpbin.org/get")
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("Status code: %d", resp.StatusCode)I put voluntarily a free proxy (from spys.ru) which had a bad uptime (maybe the proxy is dead).
So here the response is : Get "https://httpbin.org/get": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
The thing is, it waits 10 seconds to return that the proxy is bad. Why not improving the proxy testing, instead of waiting the client timeout to throw the error?
The exemple here may seem a little silly. But when I use a lot of proxies and I run a good amount of requests in multiples workers, when a proxy is down, it slow down everything by waiting the page timeout.
Maybe should we implement a timeout to test proxy before (I don't know how we could do that). And also, I propose to create a better proxy implementation (if possible), which could looks like:
// Create a proxy
myProxy := proxy.New(protocol, host, port)
// Optionally, putting a authentication
myProxy.setAuthentication(username, password)
// Then linking our proxy to our transport
transport := http.Transport{
Proxy: myProxy,
}
// Or to the client directly, which could result in gain of time
// if we don't have other settings to edit in the transport
client.setProxy(myProxy)I'd like to get a thing like this, it looks more structured than just putting our proxy as parsed URL to the transport.
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I'm running requests (using go-resty library, based on default HTTP client). This problem is I need to use proxies, some of proxies are dead (public proxies) and so, they don't answer the request. I started learning Go not a long time ago, but when I was programming in C#, the proxy handling was good, I mean :
What did you expect to see?
Proxy integration that respond better or that skips wrong/dead proxies faster.
What did you see instead?
Mainly performance losses, and when I use public proxies, Go seems not connecting to all proxies while some are really working.
(Original issue from go-resty: https://github.com/go-resty/resty/issues/320)