.NET 8: No longer build Windows-specific assets (except on Windows)

richlander commented 2 years ago

.NET has various construction challenges. One of them is that various product assets are multi-targeted for net4* targets, which we build, taking both time and space. The product would be faster to build and smaller if we did not build those targets. We should find a way to achieve that.

These assets are mostly (or entirely) unvaluable and unused. Also, today source-build targets Linux, where net4* is the least likely to be valuable.

The .NET SDK enables building net4* assets. It adds a PackageReference to Microsoft.NETFramework.ReferenceAssemblies. We can continue to offer that. It is largely unrelated to this topic.

It would be useful to do three things:

Determine the cost of these assets, in built time and/or output size.
Validate the premise that these assets are unused/unvaluable.
Can we just target net8 and netstandard*. What scenarios require targeting other TFMs?

The initial findings is that removing .NET Framework targeting would:

Reduce source size by 3GB.
Reduce build time by TBD.
Reduce build output size by TBD.

@MichaelSimons @mmitche @jaredpar @omajid @tmds

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

mmitche commented 2 years ago

We'll need to figure out where the net4* usage is coming from. I took a quick look yesterday at runtime and aspnetcore, and here is what I saw offhand:

Infrastructure - Some tools and tasks that use the desktop runtime. Some (most) of these are probably movable to core. Obviously any of these tools wouldn't work on Linux or Mac anyway. My recommendation would be to triage them and try to move all infra/tooling to core.
Tests - There were a bunch of test projects that multi-targeted. Those could go away I think
Product - I found at least one place in aspnetcore (src/SiteExtensions/Runtime/Microsoft.AspNetCore.Runtime.SiteExtension.pkgproj) that targets 462.

I am all for a full move to core.

richlander commented 2 years ago

We either need to "full move to core" or have a new feature that ignores certain families of targets. Possibly we'll need a bit of both.

mmitche commented 2 years ago

We can also make rules per platform. Basically say: Windows can use desktop targeting packs as a pre-built input, but it's not allowed on Linux. So when building for Linux, we don't build those RIDs.

It's likely we will find cases where we need to build something for desktop (e..g tooling that can't be ported). We shouldn't penalize all platforms though.

jkotas commented 2 years ago

There are many packages in dotnet/runtime that target multitarget .NET Framework. Look for $(NetFrameworkMinimum) to find the full set.

Windows can use desktop targeting packs as a pre-built input, but it's not allowed on Linux. So when building for Linux, we don't build those RIDs.

Yes, that's definitely an option. In other words, we would say that the Microsoft-built nuget packages cannot be built via offline source build.

mmitche commented 2 years ago

@jkotas Is there a reason to keep this targeting around long term?

jkotas commented 2 years ago

For as long as we want to keep supporting .NET Framework with the OOB dotnet/runtime packages. It is a complex topic. For example, we have some evidence that .NET Framework support allows people to modernize faster since they do not have to move from .NET Framework to .NET Core at once.

richlander commented 2 years ago

For as long as we want to keep supporting .NET Framework with the OOB dotnet/runtime packages.

That's goodness.

Can those packages be built from dotnet/runtime and not souce-build, at all?

mmitche commented 2 years ago

For as long as we want to keep supporting .NET Framework with the OOB dotnet/runtime packages.

That's goodness.

Can those packages be built from dotnet/runtime and not souce-build, at all?

We shouldn't do that. That keeps the runtime build around, as well as any dependency flow and build that consumes these packages. In other words, you're basically in the same situation as today, perhaps with more complexity.

I think the right call here is to remove the framework packages from the source build reference packages, and conditionalize the build so that those TFMs are not built when targeting platforms that do not allow an online restore (or need them). Basically anything but Windows.

There really cannot be independent repo official builds for anything but some isolated leaf components like the tooling.

richlander commented 2 years ago

OK. Makes sense.

Does generating the net[old] targeting packs need to be done from this repo? That can surely be a foreign pre-built. Right?

mmitche commented 2 years ago

They're already on NuGet. Just need restore them (the SDK does this automatically I think).

richlander commented 2 years ago

Right.

terrajobst commented 2 years ago

Generally speaking, we should tie multi-targeting for net4x to whether or not we offer netstandard2.0. That's because .NET Framework 4.6.1 is considered implementing .NET Standard 2.0 but it only rally works starting with .NET Framework 4.7.2.

~~Said differently: If you target netstandard2.0, you should also target net461.~~

Edit: Looks like we're talking about different things here. This is about building the .NET platform from source, which today only happens on Linux, for Linux, specifically Red Hat. I'd agree that in that context building any Windows assets is a waste of resources, which includes net4x.

ericstj commented 2 years ago

Cc @ViktorHofer We can build dotnet/runtime without those packages. That’s how the vertical builds work today. I suspect there are plenty of components up stack that depend on them though. We could build partial versions of the packages just for upstack build. What’s the thinking around these packages in general? I don’t expect source build really needs them. It doesn’t produce any NuGet packages to publish on NuGet.org so it shouldn’t need any of these packages with cross targeting builds. An alternative would be to make sure the upstack build consume transport packages wherever it was consuming a cross-targeting “official” package.

richlander commented 2 years ago

Switch your mindset to source-build being the only build. That's the underlying (and unstated) assumption of this issue.

ericstj commented 2 years ago

That’s fine. It works fine with that. We have a build leg of the product that does produce NuGet packages (some of which have cross platform joins, in addition to cross framework joins). I’m suggesting that Source build for a single distro shouldn’t need to care about those. So it’s probably better for us to think about them as different outputs that we try to keep out of the vertical builds.

mmitche commented 2 years ago

@ericstj The intention is to eliminate the cross-platform joins (specifically multi-machine joins) and make each vertical independently buildable on a single machine, and that source build is the only build producing artifacts under the SDK umbrella. The current Microsoft official build goes away entirely. The intention is to get out of the situation where we have two different methods of building altogether. What that means is that the verticals need to be able to produce the artifacts for their respective platforms. Some platforms could also produce platform-agnostic artifacts and package them up, though of course we would only ship those artifacts to NuGet.org from one of the verticals.

The goal is a single-machine vertical build for each platform that doesn't require complex orchestration that typically ends up tied to our infrastructure implementation, is robust long-term, can be built locally by developers (or other organizations, etc.). This may require some new features in NuGet (e.g. the ability to create global tool packages in a method similar to metapackages?).

This will require some rework in some areas. For instance, today workload manifests are produced in the runtime build, requiring a join among all platforms (to generate the VS insertion manifest). This is only required for VS authoring though. @joeloff's proposed solution in this instance is to note that VS insertion artifacts across all repos are simply derived and repackaged from the primary .NET artifacts (MSIs, zips, etc.). We would move generation of VS-specific artifacts into another build, which would just consume and package up the .NET artifacts with the right metadata. This is consistent with how we work with other consumers (e.g. we don't produce VSCode artifacts, docker images, corext packages, etc. directly in the build).

omajid commented 2 years ago

Hey, @terrajobst , we have some good news about:

This is about building the .NET platform from source, which today only happens on Linux, for Linux, specifically Red Hat.

There's actually many other platforms that have adapted source-build or are in the middle of doing so. For example:

Alpine Linux (testing): https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/29253
Arch Linux: https://archlinux.org/packages/community/x86_64/dotnet-sdk/
Fedora: https://src.fedoraproject.org/rpms/dotnet6.0
Homebrew (macOS): https://formulae.brew.sh/formula/dotnet

We are working with some folks to try and get this into additional distributions, including Debian/Ubuntu, as well.

richlander commented 2 years ago

FYI: This query demonstrates targeting in dotnet/runtime. I found a naked use of net48 as well, but the following pattern seems to be the primary one.

<TargetFrameworks>$(NetCoreAppCurrent);$(NetFrameworkMinimum)</TargetFrameworks>

Macros would be very easy to collapse in terms of erasing the NETFX targets.

https://github.com/dotnet/runtime/search?q=NetFrameworkMinimum

ericstj commented 2 years ago

@mmitche - what I'm saying is not disagreeing with you at all. I'm saying that instead of removing .NETFramework from the packages, you just need to think of those packages as a different vertical. Packages are immutable and making them behave differently for source build just to facilitate codeflow and never ship them is a pretty big hack. Instead we should be thinking about those packages as produced by a vertical that's capable of producing their content. For source build we avoid depending on packages that need to have more content than is necessary for a vertical product build. In this way we'll actually naturally land in a place where we clean up the vertical source builds to produce just what we need.

mmitche commented 2 years ago

@ericstj Alright, sounds good. Thanks for the clarification.

richlander commented 2 years ago

I changed the issue title to make it more general and implementation-agnostic.

tmds commented 2 years ago

+1000 for eliminating the net4* targets and getting rid of 3GB of IL sources.

Note that previously we needed these targets to build assemblies used by mono-based OmniSharp. OmniSharp has switched to .NET (Core), so we don't need them anymore.

Ideally, a source-build of .NET X.0 should require netX.0 as a target framework only.

And, the ref packages (for .NET Core, and ASP.NET Core) for that target framework should be buildable from the runtime/aspnetcore repo sources and not require separate IL dumps.

MichaelSimons commented 2 years ago

And, the ref packages (for .NET Core, and ASP.NET Core) for that target framework should be buildable from the runtime/aspnetcore repo sources and not require separate IL dumps.

@tmds - FYI, This improvement was already made in .NET 6.0. The ref packages are getting built as part of the product.

tmds commented 2 years ago

FYI, This improvement was already made in .NET 6.0. The ref packages are getting built as part of the product.

When we can, we should remove the IL sources.

Besides not targeting net4* we should also not target older versions of .NET Core.

This is what we have in ILsrc for a tarball generated from main:

24M microsoft.aspnetcore.app.ref/3.0.1
30M microsoft.aspnetcore.app.ref/3.1.10
43M microsoft.aspnetcore.app.ref/5.0.0
41M microsoft.netcore.app/2.0.0
41M microsoft.netcore.app/2.1.0
47M microsoft.netcore.app.ref/3.0.0
47M microsoft.netcore.app.ref/3.1.0
57M microsoft.netcore.app.ref/5.0.0
8.0K    microsoft.netframework.referenceassemblies/1.0.2
52M microsoft.netframework.referenceassemblies.net20/1.0.2
517M    microsoft.netframework.referenceassemblies.net40/1.0.2
268M    microsoft.netframework.referenceassemblies.net45/1.0.2
269M    microsoft.netframework.referenceassemblies.net451/1.0.2
268M    microsoft.netframework.referenceassemblies.net452/1.0.2
269M    microsoft.netframework.referenceassemblies.net46/1.0.2
272M    microsoft.netframework.referenceassemblies.net461/1.0.2
270M    microsoft.netframework.referenceassemblies.net462/1.0.2
273M    microsoft.netframework.referenceassemblies.net47/1.0.2
275M    microsoft.netframework.referenceassemblies.net472/1.0.2
275M    microsoft.netframework.referenceassemblies.net48/1.0.2
24K netstandard.library/1.6.1
33M netstandard.library/2.0.0
33M netstandard.library/2.0.3
16M netstandard.library.netframework/2.0.1-servicing-26011-01
35M netstandard.library.ref/2.1.0

MichaelSimons commented 2 years ago

Unfortunately only the .NET current ref packages are produced by the current build. That aside, I think we are on the same page as far as where we need to get to.

MichaelSimons commented 1 year ago

This work was completed in https://github.com/dotnet/source-build/issues/3014.

dotnet / source-build

.NET 8: No longer build Windows-specific assets (except on Windows) #2901