search

dotnet / arcade

Tools that provide common build infrastructure for multiple .NET Foundation projects.
MIT License
666 stars 343 forks source link

Make sure all new packages have been mirrored to dotnet-public once per year #11503

Open wtgodbe opened 1 year ago

wtgodbe commented 1 year ago

Every time we bump major version, we get a set of "new" packages, since packages like Microsoft.DotNet.Web.ItemTemplates.7.0 have the Major version in their name. This means someone needs to manually mirror these packages to dotnet-public so that further versions on nuget.org get auto-mirrored. We should have an automatic process to pick these up (along with new OOB package that might come along in a major release) so that we don't have to remember to do this manually every year.

akoeplinger commented 1 year ago

We have the same issue with e.g. https://www.nuget.org/packages/Microsoft.NET.Workload.Emscripten.Manifest-7.0.100 which is also not on dotnet-public right now.

crummel commented 1 year ago

Source-build Linux partners ran into this for the 7.0 GA release: https://github.com/dotnet/source-build/discussions/3105.

dougbu commented 1 year ago

dotnet/aspnetcore had this issue on input to our builds w/ System.Threading.RateLimiting v7.0.0.

In case these aren't in dotnet-public (sorry, didn't check), our new outputs are:

All of the above should be available on NuGet.org and in dotnet-public.

dougbu commented 1 year ago

Should mention FR already addressed our System.Threading.RateLimiting gap.

dougbu commented 1 year ago

@akoeplinger what are the new packages in 7.0.0 GA from dotnet/runtime❔

garath commented 1 year ago

In case these aren't in dotnet-public (sorry, didn't check), our new outputs are: ...

@dougbu I have just pushed these.

garath commented 1 year ago

We have the same issue with e.g. https://www.nuget.org/packages/Microsoft.NET.Workload.Emscripten.Manifest-7.0.100 which is also not on dotnet-public right now.

@akoeplinger I have just pushed all versions to dotnet-public.

MichaelSimons commented 1 year ago

Source-build Linux partners ran into this for the 7.0 GA release: dotnet/source-build#3105.

Ideally the release process would ensure all packages are mirrored prior do declaring the release complete. I consider this a blocker for source-build.

michellemcdaniel commented 1 year ago

@tkapin What do you think about adding this to the release pipeline/process?

tkapin commented 1 year ago

@michellemcdaniel From the discussion in this issue, it seems the consensus is the value is there, although we'd benefit from the work only once per year. To reason about the ROI here, could you estimate how much work is required to implement this E2E (including testing, docs, communication). Also, is there a clear way how to implement this or some investigation would be necessary first? Thx

michellemcdaniel commented 1 year ago

chrome froze and I lost my entire comment. Ok. well. Starting again.

There are 2 ways we could do it:

1) Use dotnet-migrate-packages in the release pipeline. 2) Use the various nuget package publishing code we already have in the release pipeline (the nuget.txt files for aspnet/everything else, the NetCore.NugetPublishing.ps1 script, and following the example of copy-transport-packages.txt for doing this not to nuget.org).

The first option is a bit more unknown, because that means hooking up the repo and then running the tool. The second option I think would take someone who knows the code maybe 2 days to set up and someone who doesn't maybe a week with some help from someone who knows the code. A bit more than an FR issue, but not that much work.

FWIW, I think this could go into the https://github.com/dotnet/arcade-services/issues/2447 epic.

akoeplinger commented 1 year ago

Given we only have to do this once a year maybe it'd be enough to just keep a list of these packages with the version number in the name somewhere in OneNote and have someone push these as part of normal release activities?

michellemcdaniel commented 1 year ago

An easy way would be to take the nuget.txt files and diff that with the packages already on the feed… probably. The goal of the release pipeline was to take things that were somewhat error prone (in this case, missing a package), and automate them, even if they only happen once a release

tkapin commented 1 year ago

This looks like a good entry-level task to get a hands-on experience with the release pipeline. Would it be also possible to end-to-end test the implementation? If so, it looks like a good task for a new joiner on our team.

michellemcdaniel commented 1 year ago

In the staging pipeline, we have test feeds that we create on the fly to end-to-end test our publishing code. I'm sure we could follow that pattern. @jonfortescue worked on that and between him and me, we could walk someone through the process (all the code already exists, it would just be a matter of reusing it and hooking it all up).

tkapin commented 1 year ago

Awesome, if we can test it, it's a good task for learning about the release pipeline.