search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.89k stars 4.63k forks source link

Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible. #18280

Closed clairernovotny closed 4 years ago

clairernovotny commented 8 years ago

Using the latest System.Net.Http 4.1.1 as per https://github.com/dotnet/runtime/issues/17770#issuecomment-242131706, results in an exception when starting a web app that's .NET 4.6.1:

Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.

I've emailed a repro to @davidsh

Execution plan & status

[UPDATED by karelz]

High-level plan: A. Revert HttpClientHandler implementation in net46 build of CoreFX back to using original .NET Framework HTTP stack instead of WinHTTP (WinHttpHandler) based stack. B. Revise implementation of the 8 new APIs on HttpClientHandler we introduced in 4.1.0.0 OOB package so that it works accordingly for the net46 build.

Execution plan:

  1. Validate feasibility of [A]

    • [x] a. Port HttpClientHandler from NetFX (remove WinHttpHandler build dependency for net46 build).
    • [x] b. Add APTCA (on assembly only, no security attributes should be necessary for types or methods - same as in Desktop source code).
      • [x] Run the SecAnnotate tool to verify the claim above - Result: Passed
    • [x] c. Manually test the 2 scenarios (simplified and @onovotny’s) - Result: Verified

  2. Validate feasibility of [B]

    • [x] a. Investigate the 2 remaining APIs (DefaultProxyCredentials, MaxConnectionsPerServer) which we do not know if we can implement. - Result: They are in the bucket 4.a below.

  3. Full testing of [A] implementation (cost: 1d)

    • [x] a. Make changes in master
    • [x] b. Test all ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume master packages on myget)
      • [x] Self hosting ASP.NET Core from Windows Service - validated by @annemartijn0 (here)
      • [x] Azure Storage API - validated by @karelkrivanek (here)
      • [x] Raven.Database package + usage of new HttpClientHandler - validated by @jahmai (here)
      • [x] Direct dependency on System.Net.Http - validated by @pollax (here)
      • [x] 4.6 console app depending on System.Net.Http - validated by @MikeGoldsmith (here)
      • [x] 4.6 Azure webjob (console app) with ServiceBus - validated by @chadwackerman (here)
      • [x] 4.6 Azure Batch app - validated by @chadwackerman (here)
      • [ ] Not-yet-described scenario by @dluxfordhpf

  4. Full implementation and testing of [B]

    • [x] a. Decide on design of 4 APIs (CheckCertificateRevocationList, SslProtocols, DefaultProxyCredentials, MaxConnectionsPerServer) which we can’t implement “correctly” - we have 3 choices - either throw PlatformNotSupportedException, or do nothing, or set the properties domain-wide instead of per-HttpClient-instance
      • [x] i. Implement the decision (cost: 2d)
      • [x] ii. List all libraries on NuGet (e.g. WCF?) which use the APIs we will be technically breaking, contact them
      • 4 NuGet libraries potentially affected - owners contacted - see details and tracking progress
    • [x] b. Implement 5 APIs which we know how to implement (cost: 3d)
    • [x] c. Final testing on master branch package - covered by [3.b]

  5. Ship final packages

    • [x] a. Port changes into release/1.1.0 branch
    • [x] b. Produce new package - 4.3.1
    • [ ] c. Test most of ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume 4.3.1 stable package from myget feed - see here)
      • [ ] Self hosting ASP.NET Core from Windows Service - @annemartijn0
      • [x] Azure Storage API - @karelkrivanek
      • [ ] Raven.Database package + usage of new HttpClientHandler - @jahmai
      • [x] Direct dependency on System.Net.Http - @pollax (here)
      • [ ] 4.6 console app depending on System.Net.Http - @MikeGoldsmith
      • [ ] 4.6 Azure webjob (console app) with ServiceBus
      • [ ] 4.6 Azure Batch app
      • [ ] Not-yet-described scenario by @dluxfordhpf
      • [x] KeyVault - @Vhab (here)
      • [x] SignalR - @tofutim (here)
      • [x] OwlMQ - @JoeGordonMVP (here)
    • [x] d. Publish package on nuget.org - https://www.nuget.org/packages/System.Net.Http/4.3.1

Impact of the change - Breaking changes

Here's list of technical breaking changes caused by the proposed solution. It includes workarounds for each. Note that these new behaviors are specific when running on net46 / Desktop. When you run on .NET Core, the behavior is intact.

  1. HttpClientHandler.CheckCertificateRevocationList (introduced in System.Net.Http 4.1)
    • New behavior: Throws PlatformNotSupportedException
    • Workaround: Use ServicePointManager.CheckCertificateRevocationList instead (impacts the whole AppDomain, not just single HttpClientHandler as it did in System.Net.Http 4.1-4.3)
  2. HttpClientHandler.SslProtocols (introduced in System.Net.Http 4.1)
    • New behavior: Throws PlatformNotSupportedException
    • Workaround: Use ServicePointManager.SecurityProtocol instead (impacts the whole AppDomain, not just single HttpClientHandler as it did in System.Net.Http 4.1-4.3)
  3. HttpClientHandler.ServerCertificateCustomValidationCallback (introduced in System.Net.Http 4.1)
    • New behavior: Works fine, except that the first parameter of type HttpRequestMessage is always null
    • Workaround: Use ServicePointManager.ServerCertificateValidationCallback

  4. HTTP/2.0 support (introduced in System.Net.Http 4.1)

    • New behavior: System.Net.Http (for net46 = Desktop) no longer supports HTTP/2.0 protocol on Windows 10.
    • Workaround: Target System.Net.Http.WinHttpHandler NuGet package instead.

    • Details:

      • HTTP/2.0 support is part of the new CoreFx HTTP stack which on Windows is based on WinHTTP. The original HTTP stack in .NET Framework 4.6 did not support HTTP/2.0 protocol. If HTTP/2.0 protocol is needed, there is a separate NuGet package, System.Net.Http.WinHttpHandler which provides a new HttpClient handler. This handler is similar in features to HttpClientHandler (the normal default handler for HttpClient) but will support HTTP/2.0 protocol. When using HttpClient on .NET Core runtime, the WinHttpHandler is actually built-in to HttpClientHandler. But on .NET Framework, you need to explicitly use WinHttpHandler.
      • Regardless of whether you are running using .NET Framework runtime (with WinHttpHandler) or .NET Core runtime using HttpClientHandler (or WinHttpHandler), there are additional requirements in order to get HTTP/2.0 protocol working on Windows:
      • The client must be running on Windows 10 Anniversary Build (build 14393 or later).

      • The HttpRequestMessage.Version must be explicitly set to 2.0 (the default is normally 1.1). Sample code:

        var handler = new WinHttpHandler();
var client = new HttpClient(handler);
var request = new HttpRequestMessage(HttpMethod.Get, "http://www.example.com");
request.Version = new Version(2, 0);

HttpResponseMessage response = await client.SendAsync(request);
erdtsieck commented 7 years ago

I just installed 4.4.0-beta-24913-02 into my project. It seems to help my case.

Case: Self hosting ASP.NET Core from a Windows Service running full framework 4.6.2. dependency NETStandard.Library 1.6.1 forces me to use System.Net.Http 4.3.0.

The myget feed is very slow in my experience. It took a few minutes to install System.Net.Http 4.4.0-beta-24913-02 OK https://dotnet.myget.org/F/dotnet-core/api/v3/registration1/system.net.http/index.json 82079ms OK https://dotnet.myget.org/F/dotnet-core/api/v3/registration1/system.net.http/index.json 93813ms

karelz commented 7 years ago

Thank you @annemartijn0 for confirmation and for describing the scenario!

chadwackerman commented 7 years ago

It is currently taking five to seven minutes for this page to return a result: https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

Why is Microsoft outsourcing vital infrastructure to these shady little companies and stupid little sites? You're really choosing to pin your business AND my business on a company in Belgium called Tech Tomato?

Anyway... I already had a myget feed but it wasn't pulling down this library until after I restarted Visual Studio, clicked the "update" button buried in a dialog somewhere, then restarted Visual Studio a second time. I'm currently typing this while Visual Studio 2015 attempts to restore my packages.

Update: Visual Studio is still churning but my refresh of the myget page finally appeared. It shows perhaps a dozen downloads a day of this version of the library. Meanwhile Visual Studio summarizes it as "System.Net.Http - 75.8K downloads." I've always assumed these stats were telling me the wrong thing but here's a great example of why it's not what I want. At a minimum please tell me the current version status versus the summary so I don't unintentionally become an alpha tester without explicitly opting into the madness during this absurd moment in .NET history.

chadwackerman commented 7 years ago

5 hours later and I'm still unable to try this patch:

Attempting to gather dependency information for package 'System.Net.Http.4.4.0-beta-24913-02'
with respect to project '...', targeting '.NETFramework,Version=v4.6.1'

Five minutes later...

Package 'System.Net.Http 4.4.0-beta-24913-02' is not found in the following primary source(s):
'https://dotnet.myget.org/F/dotnet-core/api/v3/index.json'. Please verify all your online package
sources are available (OR) package id, version are specified correctly.

.NET is a tire fire.

PureKrome commented 7 years ago

Easy-up there @chadwackerman . You've opted in to their beta/alpha/non production feed so this means you're ready to except problems, etc. That said, I've been using myget for ages now (paid subscription) and have very very little issues with it. So maybe .. just maybe .. it might be your computer/net-connection/whatever? (I've had some real great wins using MyGet over the last 12+ months).

.NET isn't a tire fire, right now. Sure, there's heaps of cheese getting moved around and lots of moving parts and stuff which can (and do) go wrong. But there's heaps of people doing great stuff with the RC and RTM bits released. I've personally decided to use public release stuff only - not even BETA's or RC's anymore. So my expectations are: less problems but have to wait longer. So if you're finding some of this stuff really frustrating for you - maybe just wait a bit longer for some of this dev work to hit RTM status.

The work the corefx team (and the others in the .NET-core world) have been great and an excellent welcome with respect to going on github/open-and-transparent, compared to the bad-ole-Balmer-day's, so lets all try and stay positive and helpful to the repo-maintainers. Everyone here is human and all just trying to make the world a better place : one byte at a time.

jahmai-ca commented 7 years ago

@chadwackerman It looks like the myget feed suffering right now. I'm not 100% sure how myget works, but I think if you have a dedicated host ("dotnet.myget.org") then the feeds are actually tenanted and have QOS limits.

Going here shows the package exists, but it takes a while to come up: https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

chadwackerman commented 7 years ago

@PureKrome I'm an ex-Microsoft dev manager who was asked to check this build (see above). After personally escalating this bug internally because nobody in the .NET team was even aware of it. And now they can't manage to ship me a damn binary.

I know a tire fire when I see one. I used to put them out for Microsoft for a living.

SvenAelterman commented 7 years ago

@chadwackerman I can understand your frustration with the issues with the feed. However, the name calling ('stupid little sites' and 'shady little companies') is out of order. Tech Tomato is founded/run by qualified individuals: @maartenba, MVP awardee by your former employer, and @xavierdecoster, current MS employee; in a country (which I am partial to) that encourages innovation. The name of the company is hardly relevant and that the company was founded in Belgium shows that the .NET Core team looks beyond Redmond, WA for solutions. I am looking forward to your constructive contributions to help resolve this issue.

chadwackerman commented 7 years ago

Content edited by moderator

A Code of Conduct report was submitted against some of the comments in this thread and as a result we have removed some material judged to have violated that code. If you have any questions or concerns about that then please email conduct@dotnetfoundation.org.

kwal commented 7 years ago

@chadwackerman your former position in the company does not entitle you to bash anyone or engage in name calling. In addition, the vast majority of your 'contributions' to this thread have not only been unconstructive but petty, childish and off topic. Please troll elsewhere.

martinwoodward commented 7 years ago

Hey folks - Martin from the .NET Foundation here, wanted to explain this.

I fully appreciate that the original issue raised by this thread is frustrating and that people want it fixed. I also understand some of the wider concerns around quality and delivery and the strength of feeling in general. The team is working on resolving this issue, they’ll continue to keep folks updated.

In the meantime, please keep the tone professional and refrain from making comments that could be perceived as personal attacks against others. I've been deliberately light on the edits made as I wanted to preserve some of the strength of feeling and not overly sanitize things, but please keep things civil.

chadwackerman commented 7 years ago

Has anyone else tried to contact Tech Tomato? No response to my inquiries.

Here's what it looks like if you try to do actual work sorting out weird build warnings after adding the feed:

Attempting to gather dependency information for package
'System.Net.Http.4.4.0-beta-24913-02' targeting '.NETFramework,Version=v4.6.1'
Gathering dependency information took 1.55 min

Attempting to gather dependency information for package 
'Microsoft.Extensions.Configuration.Json.1.1.0' targeting '.NETFramework,Version=v4.6.1' 
Gathering dependency information took 2.2 min

... and so on.

And this link still takes 5+ minutes to show a page: https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

That this is a) not fixed yet b) ignored by the community and c) tolerated by the .NET team as part of day to day development serves to reinforce the obvious flaws in process, tooling, culture and management that led to this bug to begin with -- plus the bug itself being ignored for months until I personally escalated it internally.

I'm looking forward to the post-mortem @karelz promised.

maartenba commented 7 years ago

Hi Chad,

That page load time is something we are looking at. The NuGet restore time is not normal. If possible can you reach out to us (MyGet) via support at MyGet.org with mention of your NuGet.exe version in use as well as a trace with the -Verbosity Detailed switch enabled? This will definitely help us in pinpointing any perf issues.

Thanks!

chadwackerman commented 7 years ago

Package Manager Console Host Version 3.5.0.1996

I'll look at getting logs from the command line as Visual Studio goes from rock solid to unstable once I add the myget.org feed. And once an error occurs pulling down a package the whole thing tips over.

quality

p.s. @karelz: tire fire.

maartenba commented 7 years ago

Can you try this command line using the latest (nightly) NuGet.exe from www.nuget.org/downloads?

Exact command: NuGet.exe install System.Net.Http -Version 4.4.0-beta-24913-02 -Verbosity Detailed

This should output all intermediate download actions as well. Thanks!

chadwackerman commented 7 years ago

@maartenba:

On the link you sent, "latest" points to https://dist.nuget.org/win-x86-commandline/latest/nuget.exe. That is the version I already have, so I don't think that is a nightly.

I downloaded the nightly package NuGet.CommandLine.4.0.0-rtm-2254.nupkg, ran nuget.exe install on it, and it failed trying to pull down files from myget.org. FYI: 1.5 seconds to return a 404 from dotnet.myget.org.

If this is not the correct way to install a nightly build or local package please inform. You can email me at this username at gmail if you prefer to take this offline.

Still happy to help but... wow. Things really should be moving in the direction of simplicity, not me troubleshooting the secondary package host for the primary package host using a nightly drop of the package manager that, despite being on version 4.0.0-rtm, somehow has five distinct manual methods for updating itself on its distribution page, each of which requires manual user intervention.

p.s. @karelz... oh, never mind. :)

maartenba commented 7 years ago

Completely agree, but logs would be very helpful in finding out where the bottleneck lies (whether secondary package host, the client itself, a crazy CDN node, cosmic rays, ...). Will contact you offline to see if we can figure this out. Really appreciate your help in this.

karelz commented 7 years ago

I have noticed myself that myget.org is pretty slow, but I was able to install the package in question successfully in "reasonable" time (on public Wi-Fi network).

OK https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.4.0-beta-24913-02 971ms
Installing System.Net.Http 4.4.0-beta-24913-02.

We should definitely look into the overall myget.org slowness, but probably outside of the scope of this issue -- it is not typical customer scenario after all. @maartenba where's the best place to discuss that? Over here (on this particular issue), let's focus how to unblock the end-to-end validation, e.g. by using some creative workarounds. Also I wonder if other folks have been blocked so badly as @chadwackerman, or if @chadwackerman's experience is extremely bad, compared to others.

@chadwackerman given that the main goal here is to validate the end-to-end scenario, I wonder if you can provide your scenario steps? Someone else here (who is not so badly blocked by myget.org usage as you are) could finish the validation in such case. That should lower your pain and wasted time on your side. Of course, assuming it is doable and doable cheaply-ish from your side.

The last resort alternative, which I'd like to avoid, is to skip your scenario validation entirely -- if we get few other end-to-end scenarios validated, we might be ok-ish (assuming the worst case scenario where @maartenba won't be able to troubleshoot your super-bad myget.org experience :().

maartenba commented 7 years ago

@karelz taking this offline with Chad, will report back here once we figure out what's going on.

karelz commented 7 years ago

Background facts:

chadwackerman commented 7 years ago

@karelz Yes, I escalated this through internal contacts on that same date 2016/12/9. I'll provide details, emails and the projects I work on privately after your postmortem and we can wrap this up offline. Perhaps in person, perhaps using alcohol.

I since verified that MyGet.org is misbehaving from multiple friend's machines on both the East and West Coasts. Mine is a recent clean install of Visual Studio 2015, no add-ins, definitely no ReSharper. The user feedback and error handling in Visual Studio is poor. Even here other users report similar lags. I'll drop this for now to free up the thread but let's not pretend that MyGet isn't borked and the whole NuGet packaging system (and NuGet's ability to update itself) doesn't faintly smell of burning rubber and isn't also a contributing factor to this bug. Both originally as part of the root cause and even today as part of trying to test and ship the fix.

maartenba commented 7 years ago

Not sure if we should keep discussing this in this thread or open another thread for it. Anyway: status report!

We're in touch with @chadwackerman via e-mail - thanks Chad for the log you sent!

On the "slowness" - preliminary profiling tells us that this is caused by the number of package versions and the impact of that fact on download size for protocol data. For example, some packages require 4 MB of JSON data to be downloaded and parsed to gather metadata. This causes a big memory usage spike in the NuGet client and the need for parsing a boat load of JSON data - definitely room for improvement there although the 4.0.0 nightlies of the client seem to behave better.

On MyGet side we'll be implementing paging for these protocol blobs to reduce download size on the protocol. May take a week for this to go live. We'll also be profiling both server and client on these requests and see if there is room for optimization on either side.

We're also looking at speeding up that page load time (but that is secondary, having install/restore work fast is priority).

chadwackerman commented 7 years ago

After hours of experimentation I was able to upgrade to System.Net.Http 4.4.0-beta-24913-01 without crashing Visual Studio. I then tried to upgrade from 24913-01 to 24913-02 and got a proper error instead of a crash.

It's 2017 and updating the nightly build of the core HTTP component for the entire system from "Monday's build" to "Tuesday's build" takes more than five minutes of wall clock time on an 8-Core i7 and requires more than 4GB of RAM.

The project in question is a webjob (a rebranded command line app) consisting of 27 lines of code.

Interestingly @karelz states that this works great for the whole .NET team and he has no problems at all sucking down the binary even while casually sipping a latte from the local Starbucks.

I was able to install the package in question successfully in "reasonable" time (on public Wi-Fi network). https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.4.0-beta-24913-02 971ms

Here's some alternative facts:

Attempting to gather dependency information for package 'System.Net.Http.4.4.0-beta-24913-02' with respect to project 'webjob', targeting '.NETFramework,Version=v4.6.1' Gathering dependency information took 4.22 min Attempting to resolve dependencies for package 'System.Net.Http.4.4.0-beta-24913-02' with DependencyBehavior 'Lowest' Resolving actions to install package 'System.Net.Http.4.4.0-beta-24913-02' Resolved actions to install package 'System.Net.Http.4.4.0-beta-24913-02' System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Newtonsoft.Json.Linq.JContainer.ReadContentFrom(JsonReader r) at Newtonsoft.Json.Linq.JContainer.ReadTokenFrom(JsonReader reader) at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader) at NuGet.Protocol.HttpSource.<>c.b__15_0(Stream stream)

richlander commented 7 years ago

Ya, can you guys please move this MyGet/NuGet problem to a new issue? It has no relationship to the original issue.

@onovotny - You might consider re-opening your issue to. This issue is now super long, so the usability of the issue has been reduced.

chadwackerman commented 7 years ago

@richlander instead of policing the thread would you instead try to install the fix and report back if it solves your specific issue? Everybody seems to be blocked by this important issue yet people would rather play GitHub ping pong than do the actual work to test the fix.

erdtsieck commented 7 years ago

Test all ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume master packages on myget)

What are the 7 scenarios? Can everyone help?

clairernovotny commented 7 years ago

@richlander I'd be happy to re-open an issue, are you just looking for me to clone this one with @karelz's updates? Not exactly sure what the ask is.

karelkrivanek commented 7 years ago

My usecase was working with the Azure Storage API, which was last working in 4.0.1-rc2-24027. It looks like this is working now. I've only done a quick test, since it took about 20 minutes to update all the packages in my projects from MyGet.

chadwackerman commented 7 years ago

@onovotny Don't reopen it. We've got momentum here and we are inches away from at least a partial resolution. The thread is overwhelming for anyone just joining now, but like any other long-running unresolved issue that reveals deeper issues, it has to be. GitHub sucks for things like this.

I verified that the new binary works compatibly on one local app. No surprise there. I then tried to cut and paste those dependencies to my webjob project and I couldn't get it to work on Azure. Webjob loads fine but it fails when triggered, unable to load System.Net.Http. This is obviously my fault and I know the drill to fix it. But I'm pretty much back where I was when this bug first opened: tweaky binding remaps and when I touch NuGet all hell breaks loose, I waste an enormous amount of time, and my project passes all tests locally yet breaks at runtime upon deploy.

jahmai-ca commented 7 years ago

So our scenarios are:

  1. A dependant (Raven.Database) package uses WebRequestHandler, which was breaking at runtime as reported in this issue.
  2. Our code uses the new HttpClientHandler type and properties.

Previously I tried the binding redirect fixes but that lead to strife, I then ended up using hacks (reflection injection, duplicating and modifying 3rd party code) to work around the issues.

I have updated to the beta and removed all the hack code, and everything (our application, end-to-end, from the browser to the database) appears to be working locally! I haven't yet tried the code on the Azure platform so I'll confirm back once that is done, but I consider this significant progress.

Also, when updating the packages I found it tolerable to use the Visual Studio Package Manager Console and use this command to update the packages (rather than add another feed config and then use the UI, which is incredibly painful):

Update-Package System.Net.Http -Version 4.4.0-beta-24913-02 -Source https://dotnet.myget.org/F/dotnet-core

This took 6 minutes 53 seconds for 20 projects to be updated.

Note that all our projects had the following binding redirect automatically generated, I did not need to fiddle with binding redirects at all:

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>
    </assemblyBinding>

Almost time for high-fives!

chadwackerman commented 7 years ago

@jahmai When I used that command line it did not update my reference from 4.0 to 4.2 nor add the necessary copy flags. Make sure those are set for System.Net.Http and dependencies and it should work fine on Azure.

pollax commented 7 years ago

Our scenario is a direct dependency on types in System.Net.Http. I tested Update-Package System.Net.Http -Version 4.4.0-beta-24913-02 -Source https://dotnet.myget.org/F/dotnet-core on one project and so far it seems to work well. That's very good news. Update Forgot to mention that binding redirects from 4.0.0.0 to 4.2.0.0 were applied automatically.

Regarding the Nuget/MyGet issues, I got the following output for this single project:

Gathering dependency information took 37.47 sec Executing nuget actions took 35.15 sec Time Elapsed: 00:01:14.7537429

Note that I'm in timezone UTC +01:00, don't know when MyGet is getting the most traffic.

maartenba commented 7 years ago

@pollax Thanks. We found the issue (see my last comment above) - client+server combination. Working on making it better.

MikeGoldsmith commented 7 years ago

I can confirm using the beta System.Net.Http library from MyGet worked for my scenario:

It took around 90 seconds for the nuget package to download from MyGet and the bindingRedirect in the app.config was applied correctly.

I'm happy to help test more scenarios if they have been described somewhere.

chadwackerman commented 7 years ago

Interesting side effect: adding the 4.4.0-beta of a Windows-only .NET library broke deployment of a .NET Core application on Linux.

"dotnet restore" is hardcoded to 60 second package retrieval times. And there's no flag to select just a specific platform like "dotnet publish" supports. So for a cross-platform library your tiny Linux worker node downloads a bunch of Windows binaries unnecessarily -- then times out and fails when it hits MyGet. Amusingly the out of memory problem that crashes Visual Studio on a 32GB machine doesn't affect a 0.75 GB Linux worker because it swaps to death instead.

Yes, I logged this elsewhere. And yes, it does relate to this bug even if you don't see it yet.

karelz commented 7 years ago

Thanks everyone helping us verify the scenarios! We got so far 5 confirmations - see detailed list with links in top-most post. I consider it good enough validation to move to next stages.

If anyone discovers any concern in the meantime (related to this specific issue), please say so ASAP. Fingers crossed.

richlander commented 7 years ago

@onovotny - It does look the issue is now progressing forward, so let's just continue with this issue. It's great to see that folks are making positive progress.

chadwackerman commented 7 years ago

@karelz You can check off my scenarios which include 4.6 command line (Azure webjob) with ServiceBus dependency and a 4.6 app with dependencies on Azure Batch. Also a third app with dependencies on AWS and Dropbox which I previously moved to .NET Core just to get away from this issue (I pulled an old version to test today).

dotnet restore problem revived at https://github.com/NuGet/Home/issues/2534, Contributor Covenant fixed backchannel, tires kicked at https://github.com/Azure/azure-storage-net/issues/372, MyGet logs sent, additional performance logs requested at https://github.com/dotnet/cli/issues/5328, and I bought a huge bag of popcorn for the postmortem discussion.

karelz commented 7 years ago

Thank you @chadwackerman for your help and confirmation! Sorry for the trouble you encountered en route. I have updated the top-most post.

karelz commented 7 years ago

From my list above:

I am preparing official info about the patch release - listing all technical breaking changes and workarounds (thanks to @davidsh for the data).

I have added info to the top-most post - see section "Impact of the change - Breaking changes" for list of technical breaking changes (4), each with a workaround.

karelz commented 7 years ago

NuGet libraries impacted by the technical breaking change (described in top-most post) -- luckily "just" 4 NuGet libraries which use any of these new APIs:

System.Private.ServiceModel_4.3.0

Consul_0.7.2.1

Octopus.Client_4.6.1

Geotab.Checkmate.ObjectModel_5.7.1701

Sorry for the inconvenience to all affected packages authors.

maartenba commented 7 years ago

On the Visual Studio / NuGet crash / swapping on Linux: the reason for this is in the way the NuGet protocol works. I have documented findings in this issue: https://github.com/NuGet/Home/issues/4448

On MyGet side we will be deploying a change after the weekend (currently in test, ETA in production Monday Feb 7) that alleviates this server-side.

maartenba commented 7 years ago

Fix on MyGet side is live. Should work fine in Visual Studio. When using NuGet.exe, make sure to use the NuGet.exe embedded in https://dotnet.myget.org/F/nuget-build/api/v2/package/NuGet.CommandLine (4.0 nightly) - the 3.5 seems to fail figuring out dependencies (but not always). Bug logged: https://github.com/NuGet/Home/issues/4512

chadwackerman commented 7 years ago

Thanks for the deep dive on this @maartenba. Never underestimate the impact even a minor tooling fix can have!

Interesting that the entire .NET team could miss both the Visual Studio crash and the NuGet problem.

I once asked an room of 80+ Microsoft devs to raise their hand if anyone was having problems setting breakpoints in the debugger. I got two hands. Compiler changed symbol format, you couldn't build the project without the latest compiler, but the debuggers hadn't updated yet.

For months nobody could set a breakpoint. On two platforms you couldn't even get a stack trace! I get called into the build lab at 1am because I'm the only other person around, get a screen full of assembly for a processor I've never even seen docs for, pull up a backtrace and the debugger crashes into the debugger.

When you change the project format while you're changing the code that parses the project format while you're dogfooding a new version of the package manager that plugs into the new version of Visual Studio -- stuff like this results. A mortal can only deal with so much change at once and this is why devs keep dropping the ball everywhere. Both us and them!

jirisykora83 commented 7 years ago

If somebody want simple powershell script for fix bindingRedirect in all app.config here it is. Probably it is not best solution but right now i have project with lot of webjobs sub-projects and it is really frustrating manually changing all file binding after updating some package.

param(
    [string]$SourceDirectory,
    [string]$Package,
    [string]$OldVersion,
    [string]$NewVersion
)

Write-Host "Start fixing all app.config"

[array]$files = get-childitem $sourceDirectory -Include app.config App.config -Recurse | select -expand FullName
foreach ($file in $files)
{
    Write-Host $file
    $xml = [xml](Get-Content $file)
    $daNodes = $xml.configuration.runtime.assemblyBinding.dependentAssembly
    foreach($node in $daNodes)
    {
        if($node.assemblyIdentity.name -eq $package)
        {
            $updateNode = $node.bindingRedirect
            $updateNode.oldVersion = $OldVersion
            $updateNode.newVersion =$NewVersion
        }
    }
    $xml.Save($file)
}

Write-Host "Done"`

Example use: ./scripts/FixAppConfig.ps1 -SourceDirectory "--project-path--" -Package "System.Net.Http" -OldVersion "0.0.0.0-4.3.0.0" -NewVersion "4.0.0.0"

georgiosd commented 7 years ago

When does this go public again? :)

karelz commented 7 years ago

Our change got into release/1.1.0 branch on Tuesday - package version 4.3.1. All packages in the branch were fliped to stable yesterday (independent effort, but helps us as well :)). @davidsh will do sanity test on myget feed (ETA: today), then we will ask for one last validation here by community (ETA: today, EOD). Once we have final validation on that build, we will push the package to NuGet. My expectation is that it will take less than a week.

We had a delay in progress and communication, because we had to convince shiproom, why this is the best and only fix. On top of that we are preparing a plan (based on shiproom feedback) to stop shipping this out-of-ban package entirely in .NET Standard 2.0 and forward all functionality to in-box Desktop framework (.NET Core functionality will be untouched) - i.e. no binding redirects will be needed if you target .NET Standard 2.0. Once I have details on impact on all scenarios, I will update this thread (in 1-2 weeks).

gulbanana commented 7 years ago

That's welcome news, and will make netstandard2.0 easier to use.

karelz commented 7 years ago

@davidsh verified the 4.3.1 package from release branch (warning: myget was very slow for him - 5 mins) Here are steps to validate:

Please try out the latest packages on the MyGet dev feed. You'll want to use the latest STABLE (not Prerelease) build of System.Net.Http.dll package - 4.3.1: https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

You can use dev feed packages via changing your NuGet package source feeds either with command line tools or Visual Studio.

Instructions:

Command-Line: Add the following in nuget.config, under element:

<add key="myget.org dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />

Visual Studio: In VS, Tools->Options->Nuget Package Manager->Package Sources -> Add, under Source, use this url, https://dotnet.myget.org/F/dotnet-core/api/v3/index.json

In either case, make sure you list the MyGet dev feed as the first feed in the list.

[EDIT] Expect binding redirect for to 4.1.1.0 in your config file:

<dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
</dependentAssembly>
karelz commented 7 years ago

I have updated top post 'Execution plan' with next steps: