Design and implement shared build infrastructure for the VMR

[ViktorHofer]

Mentioned in today's sync meeting. It would be good to define which build infrastructure should be shared and required across all repositories inside the VMR. I was thinking of the following pivots:

1. The eng/common/ layout: That directory tree is already read-only in consuming repositories and not intended to be changed.
2. The .NET SDK: The toolset SDK is the thing that is currently propagated via Arcade but picked up at different times. I.e. some repositories today prefer to not be on the latest preview. As part of the VMR consolidation, it makes so re-evaluate that option and consider requiring one version of the SDK for all repositories.
3. The Arcade SDK: Recently the Arcade SDK got more tied to the .NET SDK so that it can depend on newer features or bugfixes.

There are probably others but these are the main ones that I'm concerned about at the moment.

cc @mmitche @MichaelSimons @dotnet/product-construction

T-Shirt Size Large (includes design and implementation)

[MichaelSimons]

+1

Source-build is already requiring these today but because they way source-build worked in the past, it had to dynamically update the SDK and arcade repos use. Formalizing this will make things so much better/simpler/more understandable. One example is https://github.com/dotnet/source-build/issues/3169.

One thing to note is some repos that are included in multiple products require building with older SDK versions. For this reason we may need an option in the source-mappings to prevent the backflow of these common/shared components.

[mmitche]

I think @ViktorHofer is right on the repo infra that is important to consider. I would add to that list, though:

• NuGet.config files
• global.json files

To frame this discussion, I think it's good to call out that there are two separate copies of repo infrastructure, which may or may not be the same, but which are of equal importance:

• Repo infrastructure in context of building a repo in isolation
• Repo infrastructure in context of building within the VMR

The goals, as I see it, are as follows:

• Repo infrastructure may differ in the VMR and within a repo when in isolation (we should not force changes globally to individual repos through the VMR)
• We should avoid source edits in the VMR (this may require changes in infra or new product features)
• The VMR build must allow for arbitrary input toolsets (SDKs, arcade, and associated scripts) to be used as the build input toolset. These may not be what is specified in the global.json files.

Arcade

The first thing to note is that the eng/commmon/* scripts and the Arcade SDK are directly related and tied. While generally compatible across arcade versions, the eng/common scripts are propagated to the repos through arcade updates, based on the sha of the arcade SDK. I.e. if we built arcade SDK 9.0.0-preview.1345.5 from sha abcd, when updating arcade in downstream repos, we copy the eng/common scripts from dotnet/arcade at abcd.

The second thing to note is that more than one Arcade may be in use when building the VMR. If the VMR contains arcade (which would not necessarily be the case, as outlined in https://github.com/dotnet/source-build/blob/main/Documentation/planning/multi-sdk-band-support.md), then are two versions in play. Otherwise, there is one:

• Previously built arcade - The input bootstrap version. This is the arcade used to build repos before arcade is built from the VMR. This doesn't have to be source built in all VMR builds, though it does for Linux source build.
• Newly built arcade - This is the arcade built during the VMR build. Once built, it is used to build subsequent repos.

Side note - We could theoretically not use live-built assets to build .NET. We could always use the previously built assets. This would have some advantages and disadvantages.

Now, when we look at the VMR today, we see that there are many existing copies of the eng/common scripts, and associated arcades (as defined in the global.json of these locations):

• Common scripts
  • /src/arcade/eng/common
  • /src/<repo>/eng/common
  • /eng/common/*
• Arcade versions
  • /src/<repo>/global.json
  • /src/arcade/global.json
  • /global.json

My proposal would be something like the following:

• The arcade build packages up the eng/common scripts.
• We remove eng/common from the root of the VMR to reduce confusion
• eng/common for each repo matches what is in the isolated repo (i.e. they can vary between repos).
• The VMR build has a minimal set of scripts used to bootstrap itself enough to restore the bootstrap arcade and obtain the eng/common scripts.
• The eng/common scripts from the bootstrap arcade, are used in the root VMR build processes, as well as to build initial set of repos before we hit arcade.
• After we build arcade, we use those eng/common scripts packaged with the SDK (or in whatever package) to build the remaining repos.
• Repos could get arcade updates via the VMR, or from the isolated repo like the current subscription flow. If it's coming in via VMR, it's the arcade that the VMR built, with the associated scripts of the src/arcade sha at that point.

SDK

I think the SDK is a little simpler, based on the fact that it does not have associated scripting, and the one we use is consistent throughout the VMR build.

• The build SDK would be specified in the global.json at the root of the repo.
• Command line overrides would ignore this global.json version and use another
• We often control the version of repo's SDKs using the SDK in arcade's global.json. This would happen from the root global.json with VMR->repo flow. This would be selective based on repo (repos could choose not to get such an update).

[mmitche]

Bleh, the more I think about this, the more this feels very messy. Some examples:

• If arcade updates its tasks to target a new TFM, those new tasks will require the new runtime. Usually this is done in one commit in arcade: The Dotnet SDK gets updated alongside the arcade tasks. What does this workflow look like when the pre-arcade parts of the build are using a new SDK but old arcade, then switching to a new arcade.
• If arcade and the SDK go hand in hand, why would repos get arcade+scripts from the VMR's build output, but the SDK from its root global.json? On the other hand, if I put the SDK version in arcade's global.json like we have today, why should that be used for the whole VMR build.

[mmitche]

@jaredpar May have opinions.

[mmitche]

I have another big thought, @MichaelSimons and and others may have some thoughts on it too:

A lot of these problems seem stem from us using to build, then building and then using build-time tooling. We use bootstrap version to build a new version, then use it. But we only do this for some kinds of tooling, like arcade. For the build SDK, we use a consistent version throughout. This leads to oddities around how the build time SDK could be upgraded (or not upgraded, but some of the just-built arcade tooling could require a different SDK, yadda yadda). It's messy.

What if we just eliminated this practice altogether? We have an input toolset (arcade, sdk, etc.), and an output toolset (produced from VMR), but we do not use the arcade just-built the VMR to build the rest of the VMR. If there are other cases like this, we would eliminate them. Then, the landscape might look something like this:

• Build time Arcade and SDK - Specified in root global.json, or overridden on command line/via package props.
• eng/common scripts - Correspond to the Arcade in the root global.json. If arcade is overridden, these scripts pull from the arcade sdk package. These scripts are read only just like a regular repo today.
• Each repo has their own global.json, eng/common, and arcade SDK versions - These are not used when building in VMR mode. They just sit there. The VMR just overrides everything.
• Repos, including the VMR itself, can subscribe to arcade tooling updates (scripts, arcade version, and SDK version in the global.json) via getting updates from the VMR build.
• To update the SDK in use by the VMR, a change is made to arcade in the VMR, updating the SDK version in global.json. A build is performed, and the VMR takes that update. It gets the updated scripts, global,json SDK specification, and new arcade in the process. this only applies to the root global.json and eng/common
• To update the common scripting, a change is made in src/arcade/eng/common, the VMR is built, new arcade packages pop out, and that flows wherever. The common scripting it copied from src/arcade/eng/common to all the targets. That would also include the VMR itself.

The underlying principals here are:

• Stop doing fancy bootstrapping and stage0/stage1 style builds within the VMR build. We already pulled back on this process a bit by stopping our use of the roslyn we just built to build the rest of the VMR.
• Treat the VMR like any other repo w.r.t. to its input tooling. Keep the landscape the same as today
• When it builds, the VMR ignores the input SDK/arcade/common scripting of all repos, and substitutes the ones has checked in.

This means that the individual repos, aside from arcade, don't really need their eng/comon dirs in the VMR and they could eliminated. I suggest that we not eliminate them, though, for simplicity's sake.

[jaredpar]

Repo infrastructure may differ in the VMR and within a repo when in isolation (we should not force changes globally to individual repos through the VMR)

Don't understand what is being said here.

We should avoid source edits in the VMR (this may require changes in infra or new product features)

Is this touching on items like how we have to stomp on global.json today in order to build?

but we do not use the arcade just-built the VMR to build the rest of the VMR

Agree on this. I think it is valuable to regularly validate that we can build the VMR with the just-built arcade. Essentially that we can fully bootstrap the VMR. But I do not think it should be part of the default build experience. Doing so adds a lot of overhead and complexity for not that much gain. There are likely going to be times where we need to break compat for larger refactoring and having bootstrapping be part of the default inner build will make that more painful.

The compiler takes this overall approach (default build uses existing toolset, have a leg that validates bootstrap) and is quite happy with the results.

[MichaelSimons]

cc @NikolaMilosavljevic - this relates to the global.json and possibly the nuget.config mutations that get made during source-build.

[mmitche]

T-Shirt Size Large (includes design and implementation)

[ViktorHofer]

We need to figure out what to do with the eng/common scripts.

As we use the live Arcade SDK, we should use the corresponding eng/common files. I'm not sure how we could achieve that without doing yet another set of source edits. Maybe doing source edits is fine for now and we just always overwrite the repo ones with the ones from src/arcade?

Right now, the Windows PoC PR shows so many file changes because we updated all eng/common files manually.

[mmitche]

Yeah, agreed. I wouldn't mind doing a bit of an experiment to see how many assumptions about the eng/common and src/ layouts there is. My guess is quite a lot. This may be very difficult to achieve.

This is one reason that I was somewhat advocating for never using a live-built arcade within the build. If you don't do that then you don't have to worry about using scripts that align with the just-built arcade. Instead, eng/common is the bootstrap arcade's eng/common.

The downside to this approach is that it's harder to make cross-cutting infra changes.

[mthalman]

The downside to this approach is that it's harder to make cross-cutting infra changes.

While not ideal, we could maybe create some mini infrastructure that would allow a dev to author arcade changes and then apply those changes to the VMR as the input Arcade. Then once they have finished iterating on the changes, they could apply the Arcade changes to the arcade repo and then follow up with the rest of the VMR changes afterward. If we could make that a tight dev loop, that would be nice.

[ViktorHofer]

IMO it's useful to use the live built Arcade and the live eng/common scripts whenever possible. Here's one proposal for eng/common:

1. Add a variable $RepositoryEngineeringCommonDir to tools.ps1 and eng_common_root to tools.sh that have their default value and respect the REPOSITORYENGINEERINGCOMMONDIR environment variable if set.
2. Add an msbuild property (RepositoryEngineeringCommonDir) to the Arcade SDK with a default setting.
3. Make repositories respect that env var in native scripts and the msbuild property in the repo specific msbuild infrastructure. This would be part of the "repository contract".
4. Store the eng/common scripts in the VMR root and remove all the sub repo eng/common directories.
5. Add the REPOSITORYENGINEERINGCOMMONDIR env var to the orchestrator that points to the VMR eng/common folder and propagate that setting to the inner repos.

The same could be done for the "artifacts" folder in case we would want it to be located in the VMR root as well, i.e. VMR/artifacts/runtime/.