net/http: Transport doesn't support NTLM challenge authentication

[chen-keinan]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go 1.8

What operating system and processor architecture are you using (go env)?

  OSX darwin-amd64

What did you do?

I have send an https request to a proxy (ntlm) below request and initial response
(via wireshark)

---

Request: CONNECT www.endpoint.com:443 HTTP/1.1 Host: www.endpoint.com:443 User-Agent: Go-http-client/1.1 Location: https://www.endpoint.com Proxy-Authorization: NTLM TlRMTVNTUAABAAAAB4IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAMAA=

     **Response**
    HTTP/1.1 407 Proxy Authentication Required
    Server: FreeProxy/4.50
    Date: Thu, 20 Apr 2017 15:20:10 GMT
    Content-Type: text/html
    Transfer-Encoding: Chunked
    Proxy-Authenticate: NTLM 
    TlRMTVNTUAACAAAADAAMADgAAAAFgoECloLVra5EaVAAAAAAAAAAA
   A9KAEYAUgBPAEcAMAACAAwASgBGAFIATwBHADAAAQAOAFcASQBOA 
   ZgByAG8AZwAuAGwAbwBjAGEAbAADACYAdwBpAG4AMgAwADEAMgAu 
   wAbwBjAGEAbAAFABYAagBmAHIAbwBnAC4AbABvAGMAYQBsAAcACAD 
  Proxy-Connection: Keep-Alive
 ------------------------------------------------------------------------------------------------

The response above never reach the client, on transport.dialConn the response return status code 407 for challenge , because the response code != 200 the persist connection become nil

  -------------------------------------------------------------------------------------------
    br := bufio.NewReader(conn)
    resp, err := ReadResponse(br, connectReq) // resp.StatusCode =407
    if err != nil {
        conn.Close()
        return nil, err
    }
    if resp.StatusCode != 200 {
        f := strings.SplitN(resp.Status, " ", 2)
        conn.Close()
        return nil, errors.New(f[1]) // persist connection become nil 
    }

---

since the persist connection return nil then request is cancelled and response return as nil with error Proxy Authentication Required see --> transport.RoundTrip

 --------------------------------------------------------------------------------------------
     pconn, err := t.getConn(treq, cm) // pconn = nil
    if err != nil {
        t.setReqCanceler(req, nil)
        req.closeBody()
        return nil, err
    }
-------------------------------------------------------------------------------------------------

What did you expect to see?

I expect the response to return is it send from the proxy with status code 407

What did you see instead?

I got nil response with error: Proxy Authentication Required

Note: if I use http instead of https it works OK

This issue is blocking us from developing support to NTLM Proxy , as requests https endpoint do not return challenge from proxy

[bradfitz]

I believe this is all working as intended.

If your proxy returns an error, we don't want to return its HTTP response to the user, as that would imply the origin server replied with that.

You can use ProxyConnectHeader to authenticate to your proxy.

Let me know if I misunderstand something.

[chen-keinan]

I did used ProxyConnectHeader to send the Proxy-Authorization which is OK , at that point the CONNECT NEGOTIATE started , then proxy return 407 with Challenge header which I do not received in the client due to issue describe above. 407 is good , it is part of the NTLM proto it will enable me to continue with negotiation , same as its done with http.

since the response come as nil I cannot process the response , if the response would return with 407 and challenge (as proxy send it) it will help me, on the client side to decide on how to continue

[bradfitz]

I see. It's true we don't support authentication that takes multiple rounds.

[chen-keinan]

the same NTLM negotiation works in http , why is https different ?

[bradfitz]

Because HTTPS does CONNECT and authenticates to that to give your channel for future requests.

With HTTP you're kinda getting lucky and happen to be using the same TCP connection I suppose, but there was never explicit design or support for what you're trying to do.

[chen-keinan]

do you have any suggested workaround for this issue in the meantime ?

[bradfitz]

Implement Transport.DialContext (but leave Transport.Proxy nil) and do your own CONNECT setup before giving the Conn back to the http package?

[gogolok]

hey @chenkjfrog

I've built a modified Go version that supports NTLM proxies. I'm using it to build a Cloud Foundry CLI with NTLM support. It's WIP and I'm not sure what solution someone should target to support this in the Go base/master in the future but I hope it can somehow help you :-)

https://gist.github.com/gogolok/018443687392ea4682bd82ac8712b363

[gogolok]

I'm currently working on extending http standard library to support authentication that takes multiple rounds.

Hopefully I can present something in the near future.

[bhendo]

@gogolok Thanks for this!

I want to bring one thing to your attention.

I've been doing some testing with your patch and have found one scenario that does not work. When a request is made with the http scheme through an NTLM proxy that also intercepts the request, the proxy does not forward the request to the end server. My understanding is that the CONNECT method is meant for tunneling https requests only.

I think the reason the patch works for proxies that don't intercept is because the proxy "blindly" establishes a connection from the client to the server. But when a proxy is in intercept mode it actually maintains two connections: client<->proxy & proxy<->server

I captured some packets with WireShark to see how chrome handles http requests through an NTLM proxy. It seems to use the original request, method (e.g. GET, POST, etc), and body for each part of the handshake.

After a quick review of transport.go I think the NTLM handshake specifically for http requests might need to take place in RoundTrip rather than dialConn

I will admit this all is a bit out of my wheelhouse, so take my last statement with a grain of salt :)

[gogolok]

@bhendo Thanks for your feedback. I'm currently testing and extending my new implementation and will try to consider the intercept mode. I might come back to you then :-)

[Rots]

There's an implementation for NTLM transport at: https://github.com/Azure/go-ntlmssp

[bhendo]

It looks like this issue should be closed for the same reasons as https://github.com/golang/go/issues/22288

The dial context solution recommended in that issue: https://github.com/golang/go/issues/22288#issuecomment-337276155 still has the specific failing I described above https://github.com/golang/go/issues/20053#issuecomment-331982312

A while back I ended up writing my own transport to deal with this scenario . If anyone is looking for a transport that can handle proxies that require NTLM Proxy Authentication they are welcome to use it.

[mholt]

This might be a little different from what you're doing, but for the record: I was able to proxy requests from a web browser to a backend server that requires NTLM authentication by using a distinct *http.Transport for each downstream/client connection. This works, I think, because it forces each client connection into its own connection (or connection pool) which has a distinct auth context. Basically it does what nginx's docs say for their commercial ntlm module:

Allows proxying requests with NTLM Authentication. The upstream connection is bound to the client connection once the client sends a request with the “Authorization” header field value starting with “Negotiate” or “NTLM”. Further client requests will be proxied through the same upstream connection, keeping the authentication context.

Just thought I'd post that here in case it was helpful to anyone.

(We'll be rolling it out in Caddy 2 pretty soon, so you'll be able to look at the source code at that point. Edit: Here's the code.)