Closed clairernovotny closed 4 years ago
I happened to look at this today from another report. /cc @ChadNedzlek
This looks like it might be due to missing security attributes on the out-of-band System.Net.Http.dll compared to the inbox System.Net.Http.dll. Inbox version has AllowPartiallyTrustedCallers. So does the inbox System.Net.Http.WebRequest.dll. This means everything is treated as SecurityTransparent unless annotated otherwise.
The OOB System.Net.Http.dll is missing AllowPartiallyTrustedCallers, which makes everything security-critical. Now when the inbox WebRequest.dll loads the OOB System.Net.Http.dll it violates the security rule, since WebReqeuestHandler is transparent, but HttpClientHandler which it derives from is critical.
My repro:
ericstj is right, I have the same problem here. This is a critical issue on System.Net.Http 4.1.0 that should be repaired. I can't use this library with .net4.6.1 because it lacks consistency.
This problem is a significant pain to deal with, and in particularly makes using the Azure KeyVault client painful for my team. The only painless alternative is to downgrade to .NET 4.5.2, which is not acceptable.
Also, the workaround listed previously is insufficient. If you want to use NET 4.6.x, what we've found is you have to do the following this for it to work reliably: disable dependency checking, downgrade System.Net.Http, edit the CSPROJ and disable automatic binding redirect, modify the app.config, and typically clean/exit VS/and rebuild as I've often seen System.Net.Http in use, even for trivial projects. Here's the procedure that reliably fixes this.
<AutoGenerateBindingRedirects>false</AutoGenerateBindingRedirects>
<dependentAssembly>
<assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.1.0.0" />
</dependentAssembly>
And change it to this:
<dependentAssembly>
<assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.0.0.0" />
</dependentAssembly>
dluxfordhpf, thanks for your time explaining how you have worked this out. Redirecting to System.Net.Http 4.0 worked for me in .net4.6.1, But is still very hard to maintain (the nuget dependency). Looking forward the version that will fix this.
Redirecting could cause problems if folks are using any of the new API added in System.Net.Http 4.1.0.
in particularly makes using the Azure KeyVault client painful for my team
@ChadNedzlek had the same problem and was able to workaround it by passing in an HttpClient he created himself to the KeyVaultClient constructor. If you don't use WebRequestHandler you'll avoid the problem.
EG:
var handler = new HttpClientHandler();
// configure handler
// eg: handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => errors == SslPolicyErrors.None;
var client = new HttpClient(handler);
var kvclient = new KeyVaultClient(async (authority, resource, scope) => "foo", client);
@davidsh I think you need to put AllowPartiallyTrustedCallers
back on System.Net.Http.dll. /cc @morganbr
@dluxfordhpf Big thanks for the steps. It is temporary and we hope a fix soon but still, I can continue to work on the project !
@terrajobst This is a blocking production issue. Any idea when we can get a fix onto NuGet? Thanks!
Just ran into this myself. Would be awesome if we didn't descend into dependency hell in .Net. It's heading that way.
EDIT: Fixed with a prior comments suggestion to use the system httpclient??
new KeyVaultClient(GetAccessToken, new HttpClient(new HttpClientHandler()))
That seemed to get it...
The problem is only becoming worse as more and more nuget packages from Microsoft take a dependency on the latest version of System.Net.Http. I am probably not the only one who constantly upgrades his Microsoft nuget packages to the latest pre-release version. For example packages that no longer work for me in their latest version:
Microsoft.IdentityModel.Clients.ActiveDirectory Microsoft.TeamFoundationServer.Client Microsoft.VisualStudio.Services.Client. ....
I still don't understand why this package is still available. @terrajobst @coolcsh can we get this package taken down/fixed? It's REALLY causing issues with complex app environments. Wasted several hours trying to keep the offending package from creeping into the build. Thanks!
We are looking at binding to the System.Net.Http in the NET Framework instead of this broken thing from NuGet. I agree, this problem is ridiculous, and very expensive to deal with as you must synchronize NuGet versions between projects, prevent auto binding updates, and fix/check your app.config. What surprises me is that it’s in such a widely used assembly. Perhaps MS doesn’t really care about KeyVault that much?
It is also used by the ActiveDirectory Nugget
I have reverted some libraries from targeting .NET Standard due to this and related issues where broken NuGet packages just mess up apps that target .NET Standard. This is a sorry state of things.
Thanks for filing. We're actively looking into this. Stay tuned.
This has become a significant issue for me; we use a lot of our own nuget packages internally. And it seems nuget just will not let those bindingRedirect
s alone. Everytime we update one of our internal packages, it changes the redirect back to newVersion="4.1.0.0"
. Is there a way to stop it from doing that, or is there a fix on the horizon?
Encountered when running application over HTTPS, which worked fine over HTTP. Not sure if any other significant differences.
Workaround of setting newversion="4.0.0.0"
worked for me.
Still an issue in NETStandard 1.6.1. Why?!
System.Net.Http 4.3.0 is out. Someone tried ?
@LoulG Yep, still a problem I'm afraid.
I spoke with @terrajobst on Twitter, and he said it's actually a bigger problem, and they have like 10 people working on it now. I'm personally not sure why they are not pulling the versions of this package displaying the problem, as I thought that's what package management was for. But the next we can do at this point is wait.
@LoulG same here, I have updated all my Nuget Packages, with the release of .net core 1.1 and i got this issue
System.TypeLoadException: Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.
Firstly, I think it was due to IdentityServer/IdentityServer3.AccessTokenValidation but, with reading this issue I understand my situation t_t, I spent several hours to try to solve it
EDIT : OMG, Workaround of setting newversion="4.0.0.0" worked for me too
I try to update to 4.3 but, same issue :(((
same issue here after upgrading
Issue still present with 4.3.0.
@terrajobst can you provide any updates on this issue or any further insight into the deeper problem that @robertmclaws referenced?
Any reason why this fix works locally, but when deployed to an Azure Cloud Service, the error persists?
Sometimes packaging Azure can screw up your binding redirects, try unzipping the cspkg and see what is actually being deployed.
Here is the version of the System.Net.Http.dll
I've been working on this for a few days now. Was super happy to find this fix and in tears when deploying out to Azure.
What about the .config file containing binding redirects for the project? I actually expect it to still deploy the broken version of the assembly due to CopyLocal being true from the nuget package, but the binding redirect should ensure that the framework version of the assembly is instead loaded inside the cloud service VM.
It is!!! WTH?
`
`
Looking back into my project, the web.config has reverted as well. I'll have to pick this up again in the morning. Thanks for some leads!
I find that if you use AutoGenerateBindingRedirects and then modify any nuget package for the project, it will "fix" your previously hand-modified redirects back to the broken version. Very frustrating.
But part of the problem is, if you DON'T use AutoGenerateBindingRedirects when you add new packages, you might run into other issues too.
I've been dealing with this nonsense for over a week now while having to deploy a new version of our app. Best I can suggest is to get in the habit of checking your web.config every time you deploy.
Yeah, you have to both disable binding updates via hand edit in the CSPROJ, and then fix the .config binding redirects, AND reconfigure NuGet in the GUI to NOT update dependencies, and THEN you’re ok. Yes, this is a major PITA, I am surprised it has been out in production so long. My team has been cursing at it regularly for months now.
Unfortunately following the above still only fixes my dev environment and not azure prod. I checked the latest pub file and the web.config was properly set along with my dll that is published being the version pictured above. Unfortunately, my issues is pertaining to the Azure Search library, so this fix was proving to be promising. Any other ideas out there? At a bit of a loss. Thanks for the help.
Why is this over 2 months old? This is a huge issue. Please fix.
This is completely a stop-ship issue, it absolutely needs to get addressed.
For f#$@'s sake guys, fix this already. This is idiocy.
@suprduprmn I updated and consolidated all packages and then changed this in all the app.config and web.config files:
<bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.0.0.0" />
Only then would my web app launch on Azure and be able to make https calls to Azure services that depend on System.Net.Http. YMMV but good luck.
And @terrajobst ... is there a proper place I can formally complain about neglecting major issues like this? The happy-go-lucky rules of open source don't apply here. This is Microsoft Showstopper 101 stuff circa 1995. There's no way "the community" can help. You gotta fix it and you gotta architect tools and process to make sure it stops happening. I'm seeing stuff like this in multiple Microsoft projects and it's totally unacceptable. It's obvious that there are huge gaps in testing. Basic scenarios simply don't install or build clean let alone function properly at runtime.
I want to apologize for the XXXL long time it took us to respond to this issue. It has been 1 month since last response and the issue is opened for 3+ months. I agree that this is not acceptable for a high-impact issue like this.
It was brought up to my attention this morning and I tried to do some historical digging for last few hours. The good news is that the right folks are looking into the problem for last few weeks (maybe even longer, I didn't check all the history). Sadly, the solution is very complicated :( and that's why we do not have good answer or ETA yet (although it is not an excuse for lack of update from our side). There are couple of potential solutions, but even the experts are not in agreement yet and there are downsides to each of them. We will start daily sync ups on the issue next week, trying to get solution for the problem as fast as possible. I will post updates (hopefully with more technical details) when we have them, at least on weekly basis.
Sadly, this is the best I can do at this point (i.e. sincere apology and assurance we are and will treat it seriously and do our best to fix it ASAP). If anyone has alternative suggestions, I am all ears.
@karelz I wouldn't want to distract the team from actually fixing the problem right now, but personally I'd love to hear a post-mortem analysis on the root cause of this issue and how it made it's way through QA. This has caused major headaches and I think transparency would earn some trust.
@jahmai Understood, I am also interested in that, but I want to focus on solution first. Then we can analyze what happened why and how to prevent such mishaps in future.
Thank you for the response. I think the best solution at the moment is to deactivate any package that doesn't point back to System.Net.Http 4.0.0. There are at least 2 versions of the package that are bad. Isn't that the point of distributing this stuff through NuGet in the first place?
hugs @ ms-team
Also, just so you know, between this issue, the issues around HttpClient being so poorly designed, the issues around Microsoft.Security.Owin.Jwt being broken against current dependencies, and the state of .NET Core...
It is an incredibly frustrating time to be a .NET developer right now. Each deployment now requires 30 minutes of checking assembly binding references so my apps don't get broken in production. This is not the Microsoft I've championed for so long. I love you guys, and I don't want to be harsh at all... But it feels like some tough love is needed to restore the quality status-quo.
Whatever has to be done. Even if it means shipping a 4.3.1 that references the old 4.0 package. Please, just do it soon.
Thanks guys. FWIW, if you need to make a breaking change, please do so. We dislike this, but we’ve been living with the pain for several months, and now that we know you’re engaged, we can wait a couple more, even if we have to make some API changes.
Something is off the rails here. I've been shipping C# apps for 15 years. Ironically as higher level abstractions are being shipped by Microsoft I'm spending more and more time digging deep into internals I never previously had to worry about. YOU'RE DOING IT WRONG.
I'm not smart enough to understand why reverting the trust flags on this library is a huge deal nor do I understand why I don't have more control over that from my app anyway.
If I wanted to be surprised at runtime by random library and package-management induced errors that the compiler didn't catch, I'd write my apps in Ruby.
Quick update: We have met several times this week. We brainstormed the options and figured out funding. @CIPop is working on this issue as top priority (work transitioned from @davidsh who will be OOF starting next week for the rest of the year).
Status:
Thank you for the detailed info, we appreciate it.
What about a combination of Option 2 and re-shipping 4.3 as 5.0? Since it was a technically a breaking change, you could then ship an OOB WebRequestHandler.dll for desktop as v5.0 as well. That would let you re-implement functionality in WinHttp, deprecate properties that don't map, and gives everyone a way forward in the way that SemVer is supposed to let you. Upstream would still need to fix their code too, but thats not unexpected in a major release, and they could also just upper-bound their packages to not include 5.0.
I mean, I get the idea of wanting to ship the whole framework package groups with the same version numbers, but a) that pooch was already screwed when you guys shipped everything as 4.0 instead of following the existing Desktop versioning, and b) the actual assembly versioning is already all over the place (System.Security.Claims 4.3 package ships 4.0.2 dll, which is super-annoying when having to build binding redirects). So the damage is already done.
dotnet/corefx#3 looks like the only real root-cause solution to me.
@robertmclaws We do not think that the version change would make a difference. Most people (app and library owners) typically just "hit upgrade to latest" on their packages, they don't care how much the version changed (minor vs. major). So the breaking impact will be exactly the same. It is even worse that the "breaking" effect will come to light only when you combine multiple things. That's why this issue slipped through in the first place -- we do not have testing for that combo (and I would argue that's ok - one can't test all combinations, but we can discuss it in post-mortem later).
I don't think anyone cares too much about whole framework groups versions either. Honestly, if I believed it would help majority of customers, I would vote for just changing the numbers -- I just don't believe it would help almost at all. It would just change our message to "yeah, you're broken - that's what the version says, you just didn't realize what kind of things you agree to when taking the version", which is IMO lame excuse.
Given the difference in opinions, I am interested what others think: please use upvotes /downvotes on this post:
For versioning of breaking changes, I think 5.0 is a good idea, but I don’t feel that strongly about it.
Using the latest System.Net.Http 4.1.1 as per https://github.com/dotnet/runtime/issues/17770#issuecomment-242131706, results in an exception when starting a web app that's .NET 4.6.1:
Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.
I've emailed a repro to @davidsh
Execution plan & status
[UPDATED by karelz]
High-level plan: A. Revert HttpClientHandler implementation in net46 build of CoreFX back to using original .NET Framework HTTP stack instead of WinHTTP (WinHttpHandler) based stack. B. Revise implementation of the 8 new APIs on HttpClientHandler we introduced in 4.1.0.0 OOB package so that it works accordingly for the net46 build.
Execution plan:
Validate feasibility of [A]
Validate feasibility of [B]
Full testing of [A] implementation (cost: 1d)
Full implementation and testing of [B]
Ship final packages
Impact of the change - Breaking changes
Here's list of technical breaking changes caused by the proposed solution. It includes workarounds for each. Note that these new behaviors are specific when running on net46 / Desktop. When you run on .NET Core, the behavior is intact.
HttpClientHandler.CheckCertificateRevocationList
(introduced in System.Net.Http 4.1)PlatformNotSupportedException
ServicePointManager.CheckCertificateRevocationList
instead (impacts the whole AppDomain, not just singleHttpClientHandler
as it did in System.Net.Http 4.1-4.3)HttpClientHandler.SslProtocols
(introduced in System.Net.Http 4.1)PlatformNotSupportedException
ServicePointManager.SecurityProtocol
instead (impacts the whole AppDomain, not just singleHttpClientHandler
as it did in System.Net.Http 4.1-4.3)HttpClientHandler.ServerCertificateCustomValidationCallback
(introduced in System.Net.Http 4.1)HttpRequestMessage
is alwaysnull
ServicePointManager.ServerCertificateValidationCallback
HTTP/2.0 support (introduced in System.Net.Http 4.1)
Details:
HttpClientHandler
(the normal default handler for HttpClient) but will support HTTP/2.0 protocol. When using HttpClient on .NET Core runtime, the WinHttpHandler is actually built-in to HttpClientHandler. But on .NET Framework, you need to explicitly use WinHttpHandler.The
HttpRequestMessage.Version
must be explicitly set to 2.0 (the default is normally 1.1). Sample code: