search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.98k stars 17.67k forks source link

net/http: proxy returns 503 response but http client returns error #30560

Open nnathan opened 5 years ago

nnathan commented 5 years ago

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build551463569=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm running a squid proxy server listening on 127.0.0.1:3128 which is running as squid user with an iptables rule that drops outgoing port 53 packets emitted by the squid user. This forces squid to respond to all proxied requests using a dns name with a 503 response carrying a header X-Squid-Error: ERR_DNS_FAIL 0.

I'm also writing a go program that performs a healthcheck when squid returns a response with X-Squid-Error header by making a proxied GET request to a URL using the default http client. The proxying is enabled using the http_proxy and https_proxy environment variables.

When querying a HTTP URL the http client returns the 503 response from squid with the X-Squid-Error header.

What did you expect to see?

When querying a HTTPS URL the http client should return the 503 response from squid.

What did you see instead?

When querying a HTTPS URL the http client returns an error and nil response, with the error returning "Get https://www.google.com: Service Unavailable".

The same query to https://www.google.com using curl with the https_proxy environment variable set returns the squid 503 response.

davecheney commented 5 years ago

@nnathan can you please provide a sample program that someone else can use to reproduce the problem you are having. Thank you.

nnathan commented 5 years ago

Sure.

The program kind of relies on squid running that responds with a 503 and returns an X-Squid-Error header.

  1. The squid.conf is:
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8

acl SSL_ports port 443
acl SSL_ports port 2443
acl SSL_ports port 5222
acl SSL_ports port 8243
acl SSL_ports port 8280
acl SSL_ports port 9443
acl SSL_ports port 9445
acl SSL_ports port 9763
acl SSL_ports port 22
acl SSL_ports port 25
acl SSL_ports port 4120
acl SSL_ports port 4119
acl SSL_ports port 4122
acl Safe_ports port 4120
acl Safe_ports port 4119
acl Safe_ports port 4122
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 22
acl Safe_ports port 25
acl Safe_ports port 443
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

  1. To force DNS for squid to fail, I run: iptables -A OUTPUT -m owner --uid-owner squid -p udp --dport 53 -j DROP

  2. Now the following go program runs a webserver on port 33128 which will serve a 200 response if it didn't detect a 503/X-Squid-Error response, otherwise it will serve a 500 response with a diagnostic.

main.go:

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"
)

var listenPort = flag.Int("listen_port", 33128, "health check webserver listen port")

var healthCheckURL = flag.String("health_check_url", "http://www.google.com", "URL to perform health check on")

func main() {
    flag.Parse()

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain; charset=utf-8")

        resp, err := http.Get(*healthCheckURL)

        if err != nil {
            log.Printf("error: %v", err)
            w.WriteHeader(http.StatusInternalServerError)
            w.Write([]byte(fmt.Sprintf("error: %v\n", err)))
            return
        }

        if resp.StatusCode == 503 {
            resp.Body.Close()

            squidErr := resp.Header.Get("X-Squid-Error")

            if squidErr != "" {
                log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
                w.WriteHeader(http.StatusInternalServerError)
                w.Write([]byte(fmt.Sprintf("error detected: X-Squid-Error: %s\n", squidErr)))
                return
            }
        }

        w.WriteHeader(http.StatusOK)
        w.Write([]byte("Everying OK - did not detect any squid errors\n"))
    })

    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *listenPort), nil))
}

To run: http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go -health_check_url https://www.google.com

(change the https above to http to observe the different behaviours, whereby https will return a error on the GET and http will return a proper response)

With http://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:11 GMT
< Content-Length: 46
< 
error detected: X-Squid-Error: ERR_DNS_FAIL 0
* Connection #0 to host 127.0.0.1 left intact

With https://www.google.com:

# curl -v http://127.0.0.1:33128
* About to connect() to 127.0.0.1 port 33128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 33128 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:33128
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Mon, 04 Mar 2019 05:54:43 GMT
< Content-Length: 55
< 
error: Get https://www.google.com: Service Unavailable
* Connection #0 to host 127.0.0.1 left intact

And with https://www.google.com/ bypassing the healthcheck and using the proxy directly:

# https_proxy=http://127.0.0.1:3128/ http_proxy=http://127.0.0.1:3128/ curl -k -v https://www.google.com
* About to connect() to proxy 127.0.0.1 port 3128 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0)
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: squid/3.5.20
< Mime-Version: 1.0
< Date: Mon, 04 Mar 2019 05:56:07 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3725
< X-Squid-Error: ERR_DNS_FAIL 0
< Vary: Accept-Language
< Content-Language: en
< 
* Received HTTP code 503 from proxy after CONNECT
* Connection #0 to host 127.0.0.1 left intact
curl: (56) Received HTTP code 503 from proxy after CONNECT
davecheney commented 5 years ago

Thank you for your reply. Can you please try to reduce the program. I suggest removing the http server by moving the logic from the anonymous HandleFunc into main.

nnathan commented 5 years ago

Yep sure.

So here is a condensed version without http server cruft from earlier:

package main

import (
    "log"
    "net/http"
)

func try(url string) {
    log.Printf("trying url: %s", url)
    resp, err := http.Get(url)

    if err != nil {
        log.Printf("error: %v", err)
        return
    }

    if resp.StatusCode == 503 {
        resp.Body.Close()

        squidErr := resp.Header.Get("X-Squid-Error")

        if squidErr != "" {
            log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
            return
        }
    }

    log.Printf("Everything OK - no squid errors")
}

func main() {
    try("http://www.google.com")
    try("https://www.google.com")
}

Here is the output when running using the proxy:

$ http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 go run main.go
2019/03/04 06:02:44 trying url: http://www.google.com
2019/03/04 06:03:19 squid failure error detected, X-Squid-Error: ERR_DNS_FAIL 0
2019/03/04 06:03:19 trying url: https://www.google.com
2019/03/04 06:03:19 error: Get https://www.google.com: Service Unavailable
davecheney commented 5 years ago

I think the problem is the mechanisms between http proxying, which issues a GET with the full URL and https proxying which uses the connect verb. Can you confirm the problem only occurs when you proxy https connections via squid?

On 4 Mar 2019, at 17:05, Naveen Nathan notifications@github.com wrote:

Yep sure.

So here is a condensed version with the http server:

package main

import ( "log" "net/http" )

func try(url string) { log.Printf("trying url: %s", url) resp, err := http.Get(url)

if err != nil { log.Printf("error: %v", err) return }

if resp.StatusCode == 503 { resp.Body.Close()

  squidErr := resp.Header.Get("X-Squid-Error")

  if squidErr != "" {
      log.Printf("squid failure error detected, X-Squid-Error: %s", squidErr)
      return
  }

}

log.Printf("Everything OK - no squid errors") }

func main() { try("http://www.google.com") try("https://www.google.com") } Here is the output when running using the proxy:

$ http_proxy=http://127.0.0.1:3128 https_proxy=http://127.0.0.1:3128 ./cc 2019/03/04 06:02:44 trying url: http://www.google.com 2019/03/04 06:03:19 squid failure error detected, X-Squid-Error: ERR_DNS_FAIL 0 2019/03/04 06:03:19 trying url: https://www.google.com 2019/03/04 06:03:19 error: Get https://www.google.com: Service Unavailable — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

nnathan commented 5 years ago

This problem occurs specifically when proxying https connections which uses the CONNECT verb.

What I've learned is that there are two responses: a response for the CONNECT, and a response for the (tunnelled) proxy request.

I guess it makes sense to treat a non-successful response to CONNECT as an error, however this means a loss of information of the underlying error (which in this case is embedded as an X-Squid-Error header). For my purposes this isn't a huge issue, checking the response from a regular HTTP request is sufficient.

julieqiu commented 5 years ago

/cc @bradfitz @rsc

timruffles commented 4 years ago

Also got surprised by this. The error originates when a non-200 is received for CONNECT - https://github.com/golang/go/blob/go1.13.4/src/net/http/transport.go#L1555

Agree with @nnathan that it makes sense this is considered a sub-HTTP error, but given that the errors end up with HTTP Statuses it's very confusing.

Feels like the appropriate change here is a more informative error - e.g CONNECT received status code xxx? Or a note in the top level Do(...) docs about this case if there's a fear some users rely on this behaviour and are reading the status code out of the error.

jmillikin-stripe commented 3 years ago

I've also hit this, with a similar use case -- a proxy that blocks access to certain domains and returns additional structured error info as an HTTP header on the failed CONNECT response.

It would be nice if either Do() returned a nil error in this case, or if there could be a way to recover the original Response from the returned error.

sheetjai commented 3 years ago

I am also observing similar issue in the environment, where squid is throwing 503 error.

Is their any workaround for this issue, which anyone found?