search

dotnet / maintenance-packages

Repository that hosts packages from the .NET platform whose original home/branch is not building any longer.
MIT License
16 stars 10 forks source link

Enable maintenance-packages in source build #140

Open ericstj opened 6 days ago

ericstj commented 6 days ago

This enables source-build pass on maintenance-packages. https://github.com/dotnet/source-build/issues/4684

If there are any blockers with this we could resort back to conditioning the uptake of these packages so that SB would only see versions that are available in SBRP, but I want to try to do the better thing first.

With this change I still see the following prebuilts that I would like help resolving:

: error : 8 new pre-builts discovered! Detailed usage report can be found at /workspaces/maintenance-packages/artifacts/source-build/self/prebuilt-report/baseline-comparison.xml.
: error : See https://aka.ms/dotnet/prebuilts for guidance on what pre-builts are and how to eliminate them.
: error : Package IDs are:
: error : Microsoft.AspNetCore.App.Ref.6.0.35
: error : Microsoft.NET.Sdk.IL.8.0.0
: error : Microsoft.NETCore.App.Ref.6.0.35
: error : Microsoft.NETFramework.ReferenceAssemblies.1.0.3
: error : Microsoft.NETFramework.ReferenceAssemblies.net462.1.0.3
: error : Microsoft.NETFramework.ReferenceAssemblies.net472.1.0.3
: error : runtime.linux-x64.Microsoft.NETCore.ILAsm.6.0.0
: error : runtime.linux-x64.Microsoft.NETCore.ILDAsm.6.0.0

The ref-packs are a common problem that everyone should hit, maybe we're just missing something. For IL, we need a way to consume those outputs of runtime as a toolset dependency. I'd like some help from @dotnet/source-build to recommend how to fix these.

NikolaMilosavljevic commented 6 days ago

Are your projects conditioned to not target .NET FX in source build? That might be the reason for some of the prebuilts.

MichaelSimons commented 6 days ago

A source-build leg should be added as part of this PR. Should be easy to set the enableSourceBuild parameter on the jobs template. A side effect of this is it would easily allow me to see the prebuilt report to know where the prebuilts are coming from.

MichaelSimons commented 6 days ago

I was expecting that you would need a Version.Details.xml ProductDependency on source-build-reference-packages.

MichaelSimons commented 5 days ago

I was expecting that you would need a Version.Details.xml ProductDependency on source-build-reference-packages.

I see this is defined as a ToolsetDependency. I don't think that matters that it is a toolset vs product dependency.

I see arcade and the SBRP references are 8.0. I assume this repo is a version agnostic repo and will be referenced by multiple versions of the product? This may present some challenges if we make breaking changes to the arcade/UB/SB infra over time.

MichaelSimons commented 5 days ago

Are your projects conditioned to not target .NET FX in source build? That might be the reason for some of the prebuilts.

I took a brief look and confirmed this is the case. Source build must exclude .NET FX TFMs.

ericstj commented 5 days ago

@NikolaMilosavljevic / @MichaelSimons - what about getting the ILProj SDK and ILDASM/ILASM? These are built by runtime. I'd want to tell source build that, and let it upgrade them with the built copy on the second pass.

We exclude NetFx from packages for source build too? I didn't realize that -- I thought we'd still build that stuff since it's sprinkled throughout the product (eg: task folders). If we're actually excluding netfx then I wonder if we really want to bring this repo into source build. Most of the runtime assemblies it provides are only for older frameworks - so source build should be fine with just reference surface area. Then again - we'd need to continuously regenerate that surface area as we build new package. @ViktorHofer what do you think?

ViktorHofer commented 5 days ago

Yes, the .NET product, built from source doesn't support building .NET Framework assemblies anymore. That was a pretty big effort that Michael's team drove in the last two (?) years to reduce the source-build artifact size and build times. The .NET Framework targeting packs also got removed from SBRP.

@ViktorHofer what do you think?

We already need to add new versions of actively developed packages to SBRP, i.e. STJ: https://github.com/dotnet/source-build-reference-packages/tree/main/src/referencePackages/src/system.text.json

IIRC the plan for maintenance-packages is to only release new package versions on demand (or when a TFM that is targeted goes out-of-support). That should be way less often than its current traffic, maybe only once very two years.

To onboard this repository to the source-build graph, it needs to be added to the VMR which I would rather avoid. Honestly, it feels wrong to me to add a maintenance-only repository to it.

MichaelSimons commented 5 days ago

IIRC the plan for maintenance-packages is to only release new package versions on demand (or when a TFM that is targeted goes out-of-support). That should be way less often than its current traffic, maybe only once very two years.

To onboard this repository to the source-build graph, it needs to be added to the VMR which I would rather avoid. Honestly, it feels wrong to me to add a maintenance-only repository to it.

Onboarding maintenance-packages arose as a result of this runtime PR. The issue source build has is the dependency on the preview versions. Help me understand why this is versus the latest release of the packages? This is what causes the problem with source-build. Adding the preview versions to SBRP is not scalable in general and is against the current policy.

Note: All new packages should be for released stable versions. Adding preview/release candidate packages are for exceptional cases only and require approval from dotnet/source-build.

ericstj commented 5 days ago

These packages binaries do not ship inside the source-built product. They are needed when building intermediates that might target netstandard2.0 or older versions of .NET. The fact that we work today with SBRP is proof of this.

maintenance-packages itself is a special repo that lives outside the product. It's not part of 8.0, 9.0, or 10.0 -- it's on its own where things go after they're not part of the latest .NET. I can see it remaining separate. It's also going to lag behind the latest product's infrastructure since it's a servicing repo that's meant to ship at any time, it will stay on the LTS infra. I can see this being a problem when it needs to be consumed by the latest in development release if we try to force it into the product. It was really an open question to me if it belonged in source-build or not.

We don't intend to have the product take a dependency on pre-release versions moving forward. We're just doing it this first time to test and understand if we need to do anything before the first stable release of maintenance-packages. In the future releases from m-p will look a lot like servicing releases from a previous servicing band. IIUC those are handled by SBRP today. @MichaelSimons does this address your questions?

MichaelSimons commented 3 days ago

We don't intend to have the product take a dependency on pre-release versions moving forward. We're just doing it this first time to test and understand if we need to do anything before the first stable release of maintenance-packages. In the future releases from m-p will look a lot like servicing releases from a previous servicing band. IIUC those are handled by SBRP today. @MichaelSimons does this address your questions?

Yes that answer's my question. I would be amenable to adding the pre-release versions to SBRP for this initial setup with the understanding that we would clean them up/upgrade once we have the first stable release of maintenance-packages. This would eliminate the need to SB maintenance-package/include them in the VMR.