dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.36k stars 4.75k forks source link

Replace pkgprojs with NuGet Pack task based projects #75966

Open ViktorHofer opened 2 years ago

ViktorHofer commented 2 years ago

During the .NET 6 timeframe, the .NET libraries infrastructure team heavily invested in replacing the legacy pkgproj packaging system with NuGet Pack task based projects. The completion of that work resulted in a significant simplified servicing process as various flawed concepts that complicated publishing, RTM and servicing were eliminated:

We, the .NET libraries infrastructure team don't plan to further invest in the pkgproj packaging system and eventually want to remove it entirely in favor of the NuGet Pack task which is the official way to produce nuget packages.

This repository still relies on the pkgproj infrastructure for the producing runtime and host packages:

While these packages don't use the full set feature set of the pkgproj infrastructure, history has shown that any part of the custom packaging infrastructure continues to break RTM and servicing builds for various reasons. Recent examples that would have hindered RTM and servicing of .NET 7 are https://github.com/dotnet/runtime/commit/16314656aa9d4eb3bd3bf90f9e454638958a9cc7, and https://github.com/dotnet/runtime/commit/b966a10d938bd06e35cb121391a76cbbfe3cbf70.

I suggest that we invest in replacing the remaining set of pkgproj based projects by converting them to NuGet Pack task based projects. For the mono ones, I already started with the migration a year ago but couldn't finish it because of conflicting priorities. Even though some of the packaging projects changed, it should be possible to either revive or mimic my PR: https://github.com/dotnet/runtime/pull/57499.

Size: S-M I expect that this shouldn't take more than a week per area (coreclr, mono). Also, it's unclear if the legacy host packages need to be migrated as those aren't used anymore by the the libraries (as we now use the live host).

Related: https://github.com/dotnet/runtime/issues/49137, https://github.com/dotnet/runtime/issues/59355

cc @jeffschwMSFT @marek-safar @steveisok @dotnet/runtime-infrastructure @ericstj

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

Issue Details
During the .NET 6 timeframe, the .NET libraries infrastructure team invested in replacing the legacy [pkgproj packaging system](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Build.Tasks.Packaging) with NuGet Pack task based projects. [The completion of that work](https://github.com/dotnet/runtime/issues/53202) resulted in a significant simplified servicing process as various flawed concepts that complicated publishing, RTM and servicing were eliminated: - package index maintenance - harvesting and the cross-release dependencies that it caused - the custom packaging infrastructure that still todays relies on a different set of packaging properties than NuGet's Pack task (the common path) This repository still relies on the pkgproj infrastructure for the producing runtime and host packages: - https://github.com/dotnet/runtime/blob/d6a4f7ac145df96e21ec895476cfe0eb0b1efa9d/src/mono/nuget/mono-packages.proj - https://github.com/dotnet/runtime/blob/20dd6ea921028b3a2cc660654b65112077671fb5/src/installer/pkg/projects/host-packages.proj - https://github.com/dotnet/runtime/blob/c4c1c3aac7c42494791aaa5b791ae3641dc2561a/src/coreclr/.nuget/coreclr-packages.proj While these packages only use a limited set of the pkgproj infrastructure, history has shown that the custom packaging infrastructure continues to break RTM and servicing builds as different packaging properties are used. Recent examples that would have hindered RTM and servicing of .NET 7 are https://github.com/dotnet/runtime/commit/16314656aa9d4eb3bd3bf90f9e454638958a9cc7, and https://github.com/dotnet/runtime/commit/b966a10d938bd06e35cb121391a76cbbfe3cbf70. I suggest that we invest in replacing the remaining set of pkgproj based projects by converting them to NuGet Pack task based projects. For the mono ones, I already started with the migration a year ago but couldn't finish it because of conflicting priorities. Even though some of the packaging projects changed, it should be possible to either revive or mimic my PR: https://github.com/dotnet/runtime/pull/57499. **Size: S-M** I expect that this shouldn't take more than a week per component. Also, it's unclear if the legacy host packages need to be migrated as those aren't used anymore by the the libraries (as we now use the live host). Related: https://github.com/dotnet/runtime/issues/49137, https://github.com/dotnet/runtime/issues/59355 cc @jeffschwMSFT @marek-safar @steveisok @dotnet/runtime-infrastructure @ericstj
Author: ViktorHofer
Assignees: -
Labels: `area-Infrastructure`, `Bottom Up Work`
Milestone: 8.0.0
akoeplinger commented 2 years ago

We should also replace the .pkgproj's in these repos:

ViktorHofer commented 2 years ago

Agreed. IIRC there were a few more usages in other repositories that we don't own. I will follow-up with the respective owners of such individually. I assume that llvm-project and emsdk is owned / maintained by the Mono team? Who manages the icu repo?

steveisok commented 2 years ago

I assume that llvm-project and emsdk is owned / maintained by the Mono team? Who manages the icu repo?

We manage llvm-project (except for the objwriter fork for nativeaot), emsdk, and icu.

akoeplinger commented 2 years ago

Correct, but the objwriter fork was done by copying our project files so I'd say we just fix it all at the same time, it shouldn't be hard.

filipnavara commented 2 years ago

Correct, but the objwriter fork was done by copying our project files so I'd say we just fix it all at the same time, it shouldn't be hard.

Would there be any interest in getting the objwriter merged into the same branch (and upgrading it to LLVM 14)?

akoeplinger commented 2 years ago

That is something we're discussing as well (also because it'd simplify the source-build/VMR story), but that will likely take a bit longer.

filipnavara commented 2 years ago

That is something we're discussing as well (also because it'd simplify the source-build/VMR story), but that will likely take a bit longer.

I am willing to do some heavy lifting on that if it's desired and someone can reconfigure the DARC / Azure Pipelines once it's done.

akoeplinger commented 2 years ago

Thanks, I'll ping you once we made a decision :)