x/net/http/httproxy: Never uses proxy for localhost

[vfaronov]

golang.org/x/net/http/httproxy has a hardcoded rule whereby requests to localhost never use a proxy.

This may be a sensible default, but there should be a way to override it. There is no intrinsic reason why a proxy cannot or should not be used for localhost. My use case is that I have mocks of remote HTTP services running on localhost, and I want to use mitmproxy to debug the Go program’s traffic to/from these services.

One solution might be to use the current default unless there’s a non-empty NO_PROXY/NoProxy environment variable: then I could use some dummy value like NO_PROXY=foo.invalid. But a 100% backwards-compatible solution would require some hack along the lines of NO_PROXY=but_actually_localhost_ok or HTTP_PROXY_LOCALHOST=yes.

[bcmills]

(Attn. @bradfitz)

[bradfitz]

I'm fine moving that special case rule down further pass the explicit match/no-match checks, but let's not add a new environment variable.

[vfaronov]

@bradfitz Not sure I’m following you. The function returns true (use proxy) only if none of the NO_PROXY rules match the request. So if you just move the check past all those rules, it will still fire unavoidably for all localhost requests.

[vfaronov]

FWIW, I have found this easy to work around for my use cases (I add tsohlacol to my /etc/hosts and put it in my URLs instead of localhost), so I’d be OK with closing this issue.

[JoelPagliuca]

👋 Hi all, just wasted about half an hour trying to figure out why requests to localhost weren't respecting my http_proxy. Can this be documented somewhere if this is a permanent feature of the http.DefaultClient

[bradfitz]

Honestly I don't remember why the localhost exception exists. If somebody could dig through the history to figure that out, that'd be great. Hopefully we could just remove it. But we need to understand why first.

[vbisbest]

@bradfitz we need this so we can proxy to localhost locations. I am running web server instances in docker on my local machine for testing. I am trying to sniff the traffic using a go proxy, but it just does not work because of this localhost exception issue.

[fraenkel]

Just for reference https://github.com/golang/go/issues/1589

[gopherbot]

Change https://golang.org/cl/239164 mentions this issue: http/httpproxy: Allow localhost to be proxied

[nightlyone]

What do other http clients do in this case? It might be beneficial to at least check curl and some of the popular browsers.

Please keep in mind that a major use case for proxies are corporate settings where the settings are often not changeable by end users and the proxy exceptions contain usually some internal services already. In those settings Go tests and Go applications spawning a browser might suddenly stop working. One important application in that area is redirecting an OAuth flow to the browser from the cli or the go trace viewer.

[fraenkel]

MY examination of a few applications and runtimes show that the user must be explicit, i.e., if you don't want localhost proxied add it to the NO_PROXY list.

[AlexanderYastrebov]

Related https://github.com/golang/go/issues/33695

[gwatts]

Bitten by this myself today while trying to insert a proxy between my program and a locally running service it was connecting to. Editing /etc/hosts to add a non localhost name works, but is far from ideal.

Ignoring localhost by default is fine, but there really needs to be someway to disable that behaviour - Changeset https://golang.org/cl/239164 seemed reasonable - would be nice to see it merged

[kriswuollett]

Editing /etc/hosts with localhost.localdomain as mentioned in https://github.com/golang/go/issues/7256#issuecomment-66090979 did not work for me on macOS just now. Also think that editing a privileged file on an OS is not always practical or possible for regular users of a system.

My use case was connecting to a Kubernetes API server via kubectl for a kind cluster, which is bound to localhost on a remote host as mentioned https://github.com/kubernetes/kubectl/issues/1653.

IMO this is an expected conventions bug in golang. For example, this other usage of HTTPS_PROXY works fine:

% HTTPS_PROXY=socks5://localhost:8003 curl --insecure https://127.0.0.1:46277 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}%

when also running a proxy with a command like ssh -D 8003 -v -N remote-host.