SocketsHttpHandler fails to authenticate to proxy when doing HTTPS tunneling to server

[davidsh]

Found this bug while investigating dotnet/runtime#26397.

When connecting to an HTTPS endpoint thru an authenticating proxy server where the proxy server uses Windows authentication schemes (Negotiate or NTLM), SocketsHttpHandler will not send any credentials. This causes the final HTTP response to be a 407. This occurs whether or not the authenticating proxy server closes the initial 407 response (see dotnet/runtime#26461 for related bug).

Repro code showing an authenticating proxy and resulting in no authentication by the handler and ending with a final 407 response:

static void Main()
{
    Console.WriteLine($"(Framework: {Path.GetDirectoryName(typeof(object).Assembly.Location)})");
    Socket listener = null;

    // Start a "proxy" server in the background.
    listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    listener.Bind(new IPEndPoint(IPAddress.Loopback, 0));
    listener.Listen(int.MaxValue);
    var ep = (IPEndPoint)listener.LocalEndPoint;
    var proxyUri = new Uri($"http://{ep.Address}:{ep.Port}/");

    Task.Run(async () =>
    {
        while (true)
        {
            Socket s = await listener.AcceptAsync();
            var ignored = Task.Run(() =>
            {
                using (var ns = new NetworkStream(s))
                using (var reader = new StreamReader(ns))
                using (var writer = new StreamWriter(ns) { AutoFlush = true })
                {
                    int request = 1;
                    while (true)
                    {
                        string line = null;
                        while (!string.IsNullOrEmpty(line = reader.ReadLine()))
                        {
                            Console.WriteLine($"    [request:{request}] Server received: {line}");
                        }

                        Console.WriteLine($"    Server sending response\r\n");
                        writer.Write(
                            "HTTP/1.1 407 Proxy Authentication Required\r\n" +
                            "Proxy-Authenticate: NEGOTIATE\r\n" +
                            "Proxy-Authenticate: NTLM\r\n" +
                            "Proxy-Authenticate: BASIC realm=\"IWA_Direct\"\r\n" +
                            "Cache-Control: no-cache\r\n" +
                            "Pragma: no-cache\r\n" +
                            "Content-Length: 0\r\n\r\n");
                        request++;
                    }
                }
            });
        }
    });

    var serverUri = new Uri("https://corefx-net.cloudapp.net/echo.ashx/"); // HTTPS endpoint

    var handler = new HttpClientHandler();
    handler.Proxy = new WebProxy(proxyUri);
    handler.Proxy.Credentials = new NetworkCredential("username", "password", "domain");
    using (var client = new HttpClient(handler))
    {
        Console.WriteLine($"Doing GET for {serverUri}");
        HttpResponseMessage response = client.GetAsync(serverUri).GetAwaiter().GetResult();
    }
}

If you run this code, you'll see that the loopback proxy server only receives 1 request.

(Framework: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0-preview1-26609-02)
Doing GET for https://corefx-net.cloudapp.net/echo.ashx/
    [request:1] Server received: CONNECT corefx-net.cloudapp.net:443 HTTP/1.1
    [request:1] Server received: Host: corefx-net.cloudapp.net:443
    Server sending response

If you change the above code so that the proxy only requests 'Basic' scheme, then the loopback proxy server receives 2 requests where the second request has the 'Proxy-Authorization Basic blob' request header.

(Framework: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0-preview1-26609-02)
Doing GET for https://corefx-net.cloudapp.net/echo.ashx/
    [request:1] Server received: CONNECT corefx-net.cloudapp.net:443 HTTP/1.1
    [request:1] Server received: Host: corefx-net.cloudapp.net:443
    Server sending response

    [request:2] Server received: CONNECT corefx-net.cloudapp.net:443 HTTP/1.1
    [request:2] Server received: Host: corefx-net.cloudapp.net:443
    [request:2] Server received: Proxy-Authorization: Basic ZG9tYWluXHVzZXJuYW1lOnBhc3N3b3Jk
    Server sending response

[davidsh]

cc: @stephentoub @geoffkizer

[geoffkizer]

How is this different than dotnet/runtime#26461? The repro code above is sending Connection: close. Are you saying that even without this, it fails?

[davidsh]

How is this different than dotnet/runtime#26461? The repro code above is sending Connection: close. Are you saying that even without this, it fails?

Yes. I'll edit the repro above to make it clearer. It fails to authenticate in both 'Connection: close' / 'Proxy-Connection: close' cases or not. And by "fails", I mean it stops doing any authentication and just returns final 407 response. No exceptions are thrown.

[davidsh]

Fixed in master with dotnet/corefx#30478

[davidsh]

Fixed in release/2.1 with dotnet/corefx#30516 - will be part of 2.1.3 release.