HttpClient with custom proxy settings is returning a 407.

jennyf19 commented 3 years ago

Please see this issue in Microsoft Identity Web. Customer also included a repro w/an MVC app, and is able to repro the issue there w/their custom proxy settings.

Advice from @Tratcher is to have the HttpClient owners investigate. Let us know if you need anything from our end. Thanks.

ghost commented 3 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue meta data

Issue content:
Issue content:	Please see this [issue](https://github.com/AzureAD/microsoft-identity-web/issues/732) in Microsoft Identity Web. Customer also included a [repro w/an MVC app](https://github.com/AzureAD/microsoft-identity-web/issues/732#issuecomment-726464019), and is able to repro the issue there w/their custom proxy settings. Advice from @Tratcher is to have the HttpClient owners investigate. Let us know if you need anything from our end. Thanks.
Issue author:		jennyf19
Assignees:	-
Labels:	`area-System.Net.Http`, `untriaged`
Milestone:	-

wfurt commented 3 years ago

What OS and version of .NET do you use @jennyf19? I briefly looked that the link but not would be nice to isolate simple console app for debugging. When it fails, do you see attempt to authenticate with Wireshark?

jennyf19 commented 3 years ago

@mjnorman can you answer this: " When it fails, do you see attempt to authenticate with Wireshark?" Thank you.

mjnorman commented 3 years ago

Just to elaborate here, our use case is that we are running apps on a hosted Open Source Cloud Foundry environment with no internet access. I attempted to make the repro I posted as simple as possible so as to not convolute the repro with platform specific code.

That being said, I am testing locally on macOS 10.15.7. And the sample app is netcore 3.1. Before I posted this repro, I confirmed I was not seeing communication on the proxy, but I am starting to wonder if my local environment is clouding the issue. Currently what I see is when I first launch the app, I do see communication to/from the proxy confirmed via wireshark. However, if I sign out of the page and sign back in, I see no further communication on the proxy confirmed via wireshark, even though I can see that the browser is correctly communicating back and forth with login.microsoft.com.

If I restart the application, I again see initial communication to/from the proxy and then nothing further on signing in and signing out.

So moving to testing on the hosted environment, I am never able to get the app to log in, and am met with the exception IOException: IDX20804: Unable to retrieve document from: which is indicative of no communication on the backchannel. I am not able to use wireshark on the hosted environment, but I can SSH into the instance where this application is running and can confirm that I am able to hit login.microsoft.com using the proxy via curl -x "myproxyuri" "https://login.microsoftonline.com/"

<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.office.com/login">here</a>.</h2>
</body></html>

However what I cannot validate via wireshark is that it is actually attempting to communicate on the backchannel with the proxy it is configured for on the hosted environment.

Also if you look back through the issue linked above, you will see that using the workaround of the below early in Main, allows the backchannel to work correctly.

System.Environment.SetEnvironmentVariable(HTTP_PROXY, proxy);

wfurt commented 3 years ago

Does you proxy need authentication? ( would expect so because of 407) If so, that part is not completed on macOS -> see https://github.com/dotnet/runtime/issues/24799.

mjnorman commented 3 years ago

It does. Good to know for local testing. However the issue with OSCF would not be covered under that same scenario since it is linux based (cfslinuxfs3 -- This stack is derived from Ubuntu 18.04 (Bionic Beaver)), https://github.com/cloudfoundry/cflinuxfs3

wfurt commented 3 years ago

If you have ssh access, you can try tcpdump -> it essentially creates same capture files as Wireshark. For curl, you can use the -v flag to see the exchange. This could give us some clues.

Tratcher commented 3 years ago

How does System.Environment.SetEnvironmentVariable(HTTP_PROXY, proxy); work if the proxy requires credentials?

mjnorman commented 3 years ago

Because the uri is user:pass@host.com

mjnorman commented 3 years ago

No tcpdump unfortunately, when I say I have ssh, I can ssh to the container, and the container which is built via dotnet core buildpack, does not have tcpdump. Below is the curl -v with PII redacted.

*   Trying ip...
* TCP_NODELAY set
* Connected to ... port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to login.microsoftonline.com:443
* Proxy auth using Basic with user '...'
> CONNECT login.microsoftonline.com:443 HTTP/1.1
> Host: login.microsoftonline.com:443
> Proxy-Authorization: Basic ...
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; CN=stamp2.login.microsoftonline.com
*  start date: Oct 13 00:00:00 2020 GMT
*  expire date: Oct 12 23:59:59 2021 GMT
*  subjectAltName: host "login.microsoftonline.com" matched cert's "login.microsoftonline.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: login.microsoftonline.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: no-store, no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: https://www.office.com/login
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Content-Type-Options: nosniff
< P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
< x-ms-request-id: b58f8429-689e-40dc-b289-10fb4a100300
< x-ms-ests-server: 2.1.11198.15 - EUS ProdSlices
< Set-Cookie: ...; expires=Sun, 13-Dec-2020 19:43:07 GMT; path=/; secure; HttpOnly; SameSite=None
< Set-Cookie: ...; domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None
< Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
< Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
< Date: Fri, 13 Nov 2020 19:43:06 GMT
< Content-Length: 145
<
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.office.com/login">here</a>.</h2>
</body></html>

wfurt commented 3 years ago

This is interesting as curl sends the credentials without waiting for the 407 and since that is base auth it dumps them cleartext. But it shows proxy is happy (enough) with basic auth. Based on the port I would guess the proxy is Squid. I can Try to set something up next week.

However if you have more time there is one more trick you can play to get more insight while tcpdump is not available.

set proxy to 127.0.0.1:8080 and than ssh to your system using ssh user@host -g -L8080:PROXY_IP:3128 run your code on local machine and capture traffic on loopback on port 8080. This will connect to proxy via ssh tunnel if we are lucky.

mjnorman commented 3 years ago

I tested this as below:

Set proxy uri in the app to http://127.0.0.1:8080

SSH has to be done via cf ssh myproxy -L 8080:PROXY_IP:3128 since this is OSCF.

Ran wireshark filtering for tcp/udp port 8080 and captured no traffic when running the app locally. Also the instance that opens on my local machine connects and logs into Azure with no problems (presumably because it is not using the proxy).

wfurt commented 3 years ago

what interface did you try to sniff on? for this setup you need to capture on loopback. You can also do negative check e.g. check for traffic to final destination and check you do not see it e.g. the proxy setting is respected. (or you can close the ssh and connect should fail - if not it is not going through the tunnel and poxy)

mjnorman commented 3 years ago

My mistake, I was not capturing on loopback interface. I do see traffic through the tunnel to the remote proxy. However, when I close the tunnel, I'm still able to sign in and out of the app without issue, and no further traffic is seen on the Wireshark. Additionally, re-establishing the tunnel, I still see no further traffic on the Wireshark upon signing in and signing out of the app.

So we've proven it is at least trying initially to connect to the proxy (locally at least), although still not clear as to why it does not work when I change appsettings back to the exact same IP and port that was used and working through the tunnel forward.

wfurt commented 3 years ago

That should be sufficient to see what is going on. The second CONNECT message should have some authorization header and from that we should see whet mechanism is used. If you have same .NET versos I would expect the behavior would be the same as your deployment where you cannot get packet captures.

mjnorman commented 3 years ago

Yes I did confirm that second connect has the encoded credentials that are in the app. Do you need me to supply anything else?

wfurt commented 3 years ago

and then the authentication works? That is not clear from the packet capture picture. Aside from tunneling, that logic should be identical so I would not see obvious reason why this would not work in original setup.

Thinking about it more, setting environment will impact all HttpClient instances while setting it explicitly works only for that particular one. Is there chance that what is actually used is not the one you set your proxy on?

mjnorman commented 3 years ago

Yes the authentication through the proxy works with the above setup with the exception of the oddities once the tunnel is closed and no further traffic is seen through the proxy.

Can you elaborate on your question?

wfurt commented 3 years ago

Can you verify that in your repro attempt via tunnel ALL traffic goes through the proxy? If not, the instances where the proxy is not used would fail in your production system, right?

mjnorman commented 3 years ago

No, as I mentioned above, I only saw the initial sign in process go through the proxy. No further attempts after this when through the proxy confirmed via capture. This should mean that the first attempt on the prod system would go through proxy, but this is not the case.

Also the only traffic that should be going through the proxy is the backchannel for AzureAd. as that is the only thing it is configured for, and the only traffic destined for external.

wfurt commented 3 years ago

so it works everywhere except the place you cannot debug, right? Did you try production with simple console app using just HttpClient and nothing else?

As a last shot, can you dump and check all Environment.GetEnvironmentVariables? and if name is used for the proxy host what addresses it resolves to?

mjnorman commented 3 years ago

Well when you say it "works", I would not consider the fact that while testing locally only the initial backchannel traffic out of the app goes through the proxy and any further backchannel traffic does not, as being considered "working".

The repro is a fresh webapp with azuread and htpclient for backchannel since that is my use case.

Also just to be clear, for the repro I am hard coding the test proxy url and creds, not retrieving them from environment variables.

wfurt commented 3 years ago

ok. for me, "works" means HttpClient can successfully authenticated to proxy and pass traffic. What I would like to do is to separate HttpClient from web app and test it in isolation. and possibly collect more Information about the production setup.

mjnorman commented 3 years ago

So then how do you explain the fact that closing the tunnel does not stop the app from working? If the backchannel is configured to use the proxy, and I shut down the path to the proxy, shouldn't the app break?

Can you advise what kind of additional repro you would like me to setup, and I'm happy to do this.

Tratcher commented 3 years ago

@mjnorman that depends on what needs to be fetched over the backchannel. In some configurations only the initial OpenIdConnect token signing key need to be downloaded, and those are cached for 24 hours.

mjnorman commented 3 years ago

@Tratcher makes sense, I would have expected to see more traffic but if it's only initial, then everything else is between client and MIcrosoft directly.

@wfurt I am happy to do a console app, however keep in mind the test scenario is a web app on CF. Not sure that is doable without it being a web app and Kestrel, etc? Just let me know how you want to set up the repro and I'm happy to do it.

wfurt commented 3 years ago

I understand that is not the end goal @mjnorman. At this point I'm trying to untangle as much as possible and validate small chunks. Let's start with your repro:

var handler = new HttpClientHandler(
                    {
                        UseProxy = true,
                        Proxy = new WebProxy()
                        {
                            Address = new Uri($"host:port"),
                            UseDefaultCredentials = false,
                            Credentials = new NetworkCredential()
                            {
                                UserName = username,
                                Password = password
                            }
                        },
                        PreAuthenticate = true //have tried with and without this
                    });
var client = new HttpClient(handler);
var result = await client.GetAsync(someExternalURL, default);
Console.WriteLine(result);

if you have SDK on the box just build it there or if not, publish self-contained app.

Now, if this fails, the problem clearly is in HttpClient or the environment. You can use the tunnel test again to rule out one or the other.

If this works that would verify the HttpClient can authenticate and work with your proxy. There can still be some issue with the client but it is also possible that there is something wrong with how the handler is used for example not using this handler when it should or something is changing the setting for what ever reason.

I hope it makes some sense.

mjnorman commented 3 years ago

Ok So I did confirm that this works locally, and on CF. The app crashes from a CF perspective because it is not listening to web requests, but it still spits out the result from the GetAsync using www.google.com as you can see in one of the Via headers there is the Squid Tag.--

   2020-11-19T15:13:26.22-0600 [CELL/0] OUT Starting health monitoring of container
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT {
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Date: Thu, 19 Nov 2020 21:13:26 GMT
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Cache-Control: max-age=0, private
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Server: gws
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   X-XSS-Protection: 0
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   X-Frame-Options: SAMEORIGIN
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Set-Cookie: 1P_JAR=2020-11-19-21; expires=Sat, 19-Dec-2020 21:13:26 GMT; path=/; domain=.google.com; Secure
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Set-Cookie: ---; expires=Fri, 21-May-2021 21:13:26 GMT; path=/; domain=.google.com; HttpOnly
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Accept-Ranges: none
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Vary: Accept-Encoding
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   X-Cache: MISS from 1a6ad60c-7897-4ee0-a1e2-1ec7ced952cb
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Transfer-Encoding: chunked
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Via: 1.1 1a6ad60c-7897-4ee0-a1e2-1ec7ced952cb (squid/4.7)
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Connection: keep-alive
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Expires: -1
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT   Content-Type: text/html; charset=ISO-8859-1
   2020-11-19T15:13:27.06-0600 [APP/PROC/WEB/0] OUT }
   2020-11-19T15:13:27.07-0600 [APP/PROC/WEB/0] OUT Exit status 0

karelz commented 3 years ago

OK, so HttpClient works just fine. Something else is interfering in the app. I would suggest to try similar minimal repro with just HttpClientFactory (i.e. one more layer closer to your app). Will that work? If yes, then you need to play with your app to see which piece of code breaks it and then focus on that ...

karelz commented 3 years ago

Does not seem to be actionable with current info. Closing. Feel free to reopen when there is more info available.

dotnet / runtime

HttpClient with custom proxy settings is returning a 407. #44630