How to default to WinHTTP proxy not the WinINet for a .Net core app

[karelz]

From @rohitshrivastava04 on October 11, 2017 10:50

Issue Title

How to set proxy settings for a .Net Core application

General

We have a .Net Core Windows Service which makes few calls to AWS or other internet services. For this, it needs to use proxy settings. There are two options either we configure each call to use proxy server or set proxy setting machine wide. Setting WinHTTP proxy configuration is easier than WinINet. However .net core process uses wininet as default proxy policy. How can I set it to default to WinHTTP than WinINet for the whole app rather than changing in each WinHttpHandler?

Copied from original issue: dotnet/core#1023

[davidsh]

I'm correctly what I previously said before.

If you use HttpClientHandler directly, it uses WinHttpHandler in the "Wininet" style proxy setting mode. This matches the behavior of .NET Framework.

However, if you use WinHttpHandler directly, you can use WinHTTP-style proxy settings:

var handler = new WInHttpHandler();
handler.WindowsProxyUsePolicy = WindowsProxyUsePolicy.UseWinHttpProxy;
var client = new HttpClient(handler);

And the default setting for WinHttpHandler is UseWinHttpProxy.

[rohits-dev]

Yes, but to set it I have to change everywhere we have used HttpClient. we are also using AWS SDK, Serilog to log into elasticsearch. Other option would be to set explicitly the proxy setting everywhere which I want to avoid.

[alexzaytsev-newsroomly]

Is that an option on any platform? e.g. back in full .net WebRequest.DefaultWebProxy was pretty much good to go...

[davidsh]

Using the machine-wide proxy settings for WinHTTP is very Windows platform specific. And in particular, it is specific to using the WinHTTP stack which is only used within WinHttpHandler. On .NET Core, WinHttpHandler is used as the underlying stack for HttpClientHandler.

On .NET Framework, WebRequest.DefaultWebProxy initially points to proxy settings on the Windows machine that are "Wininet-style", i.e. Internet Explorer registry settings, etc. This property can also be set to some other IWebProxy based custom class.

In general, the other way to use WinHTTP proxy settings in via WinHttpHandler class.

[alexzaytsev-newsroomly]

@davidsh not really. Linux has some standards where HTTP_PROXY environment variables could be respected.

Also, using WinHttpHandler requires refactoring of all library code, including that in third party nuget packages, whereas the approach i mentioned above (what you referred to as wininet-style proxy) the settings got propagated for every assembly loaded into the app domain.

Just looking at how this can be achieved in .net core.

[josephwoodward]

@alexzaytsev-newsroomly Did you ever find a solution to this?

[davidsh]

@davidsh not really. Linux has some standards where HTTP_PROXY environment variables could be respected.

Based on this feedback and issue dotnet/corefx#29934, we will probably implement using these environment variables on Windows as well as Linux. On Windows, the environment variables, if present, would override any "WinINet", i.e. Internet Explorer proxy settings.

Using these environment variables would provide a similar per-machine proxy setting that WinHTTP proxy settings provide (i.e. with netsh commands). But environment variables would work cross-platform.

cc: @karelz @wfurt

[wfurt]

There is always option to set proxy explicitly on HttpHandler, right @davidsh? That allows to use what ever app write wants to.

Environment variables used to work on Windows as well in early phases SocketHttpHandler for testing but than we roll it back as old handler did not work that way.

References to WebRequest.DefaultWebProxy keeps coming back as deja vu. Is that something we should address?

[davidsh]

There is always option to set proxy explicitly on HttpHandler, right @davidsh?

Yes, apps can explicitly set .Proxy property. We discourage the use of WebRequest.DefaultWebProxy property because it is legacy and tied more with legacy HttpWebRequest APIs.

The request here, though, is to use a machine-wide setting for the "default" / "system" proxy settings. This implies not having to write any code nor set the HttpClientHandler.Proxy property.

So, I think we should get the environment variable stuff working on Windows. Those environment variable should take precedence over the other Windows proxy settings (IE settings).

Environment variables used to work on Windows as well in early phases SocketHttpHandler for testing but than we roll it back as old handler did not work that way.

It's ok if this doesn't work on WinHttpHandler (or Desktop either). It just has to work in SocketsHttpHandler which is the new default for .NET Core 2.1.

[tebeco]

Hello @davidsh,

TL;DR : dotnet restore --force is still broken because it does not do anything after the first 407 CONNECT, details here :

If i manually setup the HttpClientHandler without disabling HttpClientHandler it works fine with our NTLM Proxy (thx A LOT for https://github.com/dotnet/corefx/pull/30516) :

            var handler = new HttpClientHandler();
            var client = new HttpClient(handler);

            handler.Proxy = new WebProxy("http://corpo-stuff:1234");
            handler.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials;

            return client;

There's issues with approach like this, let's take the example of AzureKeyVaultConfiguration for AspNetCore, i can pass-in an HttpClient so i would have to build it on my side and it would work. No idea if CosmoDb offer the same API

The real deal is when you do not have access to the code itself :

• we use Vsts in Hybrid mode, some agent are in Azure, some are "OnPremise", hopefully i see this : https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/proxy?view=vsts&tabs=windows
• Vscode Omnisharp it developped in netCoreApp, and i have to specify my password in clear in VsCode settings (and that is ugly for obvious reason)
• Using HTTP_PROXY is a total NO GO, as i would have to deal with NO_PROXY (most of our process try to reach LAN server and it would try to use the proxy => 502 badgateway)
• It should always try it use handler.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials; by default before asking for a user and password
• As dotnet restore --force will try to reach LAN feed and internet feed, it is not funny, also it ignores Windows Proxy for example

The Proxy use Automatic Pac script, after reading it i understand the script, we have 3-4 underlying proxy adress. So the provious code won't work. In FullFx there was an api like GetProxyUrl("someServerYouWantToReach") that returns the URL of the proxy you would have to inject instead of head-coding handler.Proxy = new WebProxy("http://corpo-stuff:1234");

What is the recommendation today ? before we could just add/edit foo.exe.config and add a Section telling the runtime please go read windows settings and use default creds Is there a way in the SocketHandler to add something like :

if(PlaterformDetection.IsWindows)
{
  var proxyUrl = GetProxyUrl("someServerYouWantToReach");
  // snipped...
}

Maybe it does already exist and we missed it completly ?

The main problem is that we would have to go through all third part and ask them to be able for us to feed them with the proxy. It was possible in FullFx to do that directly form config file so any tools developped that forgot to think about proxy would works, now it cannot

[tebeco]

> dotnet new console -n TestProxyDotnetRestore
> cd .\TestProxyDotnetRestore\
> dotnet new nugetconfig  # make sure that <packageSources> does <clear /> and only add https://api.nuget.org/v3/index.json
> dotnet nuget locals -c all

Start Fiddler,

> dotnet add package Polly

Look at fiddler : 3 exact same Request / reponse Request Raw :

CONNECT api.nuget.org:443 HTTP/1.1
Host: api.nuget.org:443

Response Raw :

HTTP/1.1 407 authenticationrequired
Date: Thu, 13 Sep 2018 07:49:59 GMT
Content-Type: text/html
Cache-Control: no-cache
Content-Length: 1873
Proxy-Connection: Keep-Alive
Proxy-Authenticate: Negotiate
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="McAfee Web Gateway"
Proxy-Support: Session-Based-Authentication

(... snipped the full HTML dump ...)

There's not retry after the initial CONNECT Usually the client should send another request with Proxy-Authorization: Negotiate SOME_HASH_HERE

This part seems to be missing

The same behavior happens using a new C# program running with dotnet 2.1.402 :

using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;

namespace ProxyNtlm
{
    public class Program
    {
        static async Task Main(string[] args)
        {
            //AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
            var program = new Program();
            await program.RunAsync();

            Console.ReadLine();
        }

        public async Task RunAsync()
        {
            try
            {
                using (var httpClient = CreateSimpleHttpClient(false))
                {
                    var response = await httpClient.GetStringAsync("https://corefx-net.cloudapp.net/Echo.ashx");
                    Console.WriteLine(response);
                }
            }
            catch (Exception ex)
            {
                Console.ForegroundColor = ConsoleColor.Red;
                var tabCount = 0;
                while (ex != null)
                {
                    for (var i = 0; i < tabCount; i++)
                    { Console.Write('\t'); }

                    Console.WriteLine(ex.Message);

                    ex = ex.InnerException;
                }

                Console.ForegroundColor = ConsoleColor.Gray;
            }
        }

        public HttpClientHandler CreateHttpClientWithHandler()
        {
            return new HttpClientHandler();
        }

        public HttpClient CreateSimpleHttpClient(bool withCredsInHandler)
        {
            if (!withCredsInHandler)
            {
                return new HttpClient();
            }
            var handler = new HttpClientHandler();
            var client = new HttpClient(handler);

            handler.Proxy = new WebProxy("http://corpo-proxy:8080");
            handler.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials;

            return client;
        }
    }
}

[glucaci]

@davidsh is this feature somewhere on the Roadmap? Can this be implemented as middleware instead of environment variables?

Thanks!

[davidsh]

@davidsh is this feature somewhere on the Roadmap?

Which exact feature are you referring to? Right now, you can use WinHttpHandler directly and select the ProxyUsePolicy that uses the 'WinHttp' style of proxy config (i.e. from 'netsh' commands). But we don't plan to make this a general feature for HttpClientHandler. That now uses the cross-platform SocketsHttpHandler stack.

cc: @karelz

[karelz]

This issue transformed into something different along the way. I am not clear what actually it tracks anymore. There have been good points about inability to set default proxy application-wide in https://github.com/dotnet/corefx/issues/24574#issuecomment-420722985, especially for code from 3rd party libraries. I plan to bring it up and discuss our options, but it is not on roadmap yet.

[tebeco]

@karelz thx i would really love to see this moving in one way or another so far if you look for answer the only one is HTTP_PROXY, this is a big no go

Aspnetcore has middleware and IXxxFeature it seems like dotnet tried a similar toggle feature with AppContext.SetSwitch

i would really love to see CoreFx with something like IRuntimeFeature<IHttpResquestHandlerFeature> with a kind of factory and netcoreapp2.x would probably avoid the wonderfull private ctor + reflection instanciation on it if it is enabled.

that way any HttpClient created by existing and already released thrid part would be injectable with custom handler (just as it work today)

This way

• we (community) could port the acces to Registry to read the proxy on windows
• make a plugable interrace for all existing third part
• you (corefx maintainer) would “just” change httpClient creation to resolve that list of handler

the worst issue for httpHandler resolution is that the order of the handlers should not be like middleware. i mean it cannot be the same order as it was plugged since it would break existing third part that are injecting handler next to the ctor/call the pluggable handlers should be resolved more “lazily”

[tebeco]

@karelz also if you prefer open a dedicated issue on the last pointed subject / direction why not

please keep in mind that dotnet restore is still broken and does not work behind proxy and we cannot use HTTP_PROXY with NO_PROXY as it lack the “dynamic resolution”

[glucaci]

Which exact feature are you referring to? ...

@davidsh I want to be able to do in ASP.NET Core the same thing that <defaultProxy> dose in web.config. That means all the calls that are made from the app to go through a proxy.

The problem is that I don't want to manipulate every HttpClientHandler because maybe I don't have access to it. For example I'm using the Azure-ServiceBus lib and those HTTP calls to Azure should go also through the proxy.

I want to be able to configure this proxy application-wide, that means, all the HttpClients that are created by the app or by other third-party libraries to use this proxy.

Thanks!

[tebeco]

https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.features.ifeaturecollection?view=aspnetcore-2.1

i would very love to see the same for the runtime itself.

[karelz]

@tebeco thanks for the link. Let's separate solution discussions from the problem though. There are many ways how this can be solved.

[glucaci]

@karelz I saw that is a milestone set on this, can you share some news how this will work or what is the status?

It's a very high issue for us and I think for all that are using On-Premise ASP.NET Core and want to reach public endpoints only through a proxy. Currently we are forced to target .NET Framework in order to use this feature.

Thanks

[karelz]

@glucaci we do not have update yet on this, but it is part of our larger 3.0 investment into Enterprise scenarios (proxies and authentication). This is a top issue for us beside HTTP/2 larger investment and 2 other painful issues I'd like to address in 3.0.

[davidsh]

Will be tracked by dotnet/corefx#36553