Build KoreBuild

[natemcmaster]

Goals:

• reduce dependency on downloading build scripts on the fly from github (example: when github is down, we are completely unable to build.)
• remove the need for preflight restore
• make builds more reproducable
• make it easier to pin versions of korebuild
• korebuild auto-updates

Implementation ideas:

• move korebuild to aspnet/buildtools
• "build" means gather together all task assemblies, loggers, scripts, etc and produce a tar.gz/zip file. Put it on a blob store.
• bootstrapper will change to download the blob. We can use a 'latest.txt' file on the blob store to keep the 'floating' behavior and we can detect when korebuild is outdated.
• blob store would have 'channels' so we can pin patch/servicing branches and still make updates to korebuild if needed. We currently use branches on this repo to keep different versions of KoreBuild.

cc @anurse @pakrym @pranavkm

[Eilon]

I will set up a short mtg to discuss.

[natemcmaster]

Notes for our meeting today

Background: current behavior

1. build.ps1 download https://github.com/aspnet/KoreBuild/archive/dev.zip
2. Extract to $repo/.build
3. Execute .build/KoreBuild.ps1
4. .build/KoreBuild.ps1 executes
   1. .build/dotnet/dotnet-install.ps1 to restore runtimes and SDKs
   2. dotnet msbuild /t:Restore /p:PreflightRestore=true to download task packages and reference assemblies
   3. dotnet msbuild KoreBuild.proj which runs the main lifecycle (build/test/verify)

Note: replace .ps1 with .sh for Linux builds

Versioning

• KoreBuild "floats" in a way. Because its download URL is per-branch, it always gets the latest source.

  https://github.com/aspnet/KoreBuild/archive/dev.zip https://github.com/aspnet/KoreBuild/archive/release/2.0.0-preview2.zip

• KoreBuild floats PackageReference versions within it's PreflightRestore step.

    <PackageReference Include="NuGetPackageVerifier" Version="2.0.0-*" />
    <PackageReference Include="Internal.AspNetCore.BuildTools.Tasks" Version="1.0.0-*" />

Problems

AutoUpdate is not possible When extracting the GitHub source archive, the bootstrapper does not have access to the "version" (i.e. commit hash) of the code it is using. This means we cannot reliably auto-update the $repo/.build folder.

Version coherence PackageReferences within KoreBuild's code are not guaranteed to match KoreBuild's code, and may change when new packages become available. This may mean long-running builds can end up with newer packages being installed half-way through a build.

Pinning is manual and requires source changes Pinning KoreBuild to exact version requires source code changes. The bootstrapper must change its download url in build.ps1, and korebuild scripts in this repo must pin their PackageReference versions.

[Eilon]

Idea is approved.

We should look at some of the exact details, such as whether to use a MyGet feed, blob storage, etc.

[natemcmaster]

Some exact details:

1. Azure storage blob

   I'm currently thinking a blob store will be easier to work with. If we use MyGet, we would need a NuGet client to acquire KoreBuild. That's easy on Windows with NuGet.exe, but running NuGet.exe on Linux would require Mono. AFAIK the only alternative is to write the curl + JSON code manually in bash to immitate what NuGet.exe does. Eww.

2. Move KoreBuild code to aspnet/BuildTools. This will make it easier to ensure the console tools KoreBuild bundles use a .NET Core runtime that aligns with what KoreBuild will install.

3. Incrementally update repos. As long as build.ps1/sh still builds the default lifecycle, we don't have to upgrade all at once.

[analogrelay]

Azure storage blob

Really agree here. I tested out MyGet and found this same issue. Azure Blobs are low-impact, low-cost and easy to use. Blobs support Etags automatically so we're easily able to use our auto-update mechanism.

[natemcmaster]

An additional detail:

• I don't plan to implement auto-update as a part of this work. There are open questions about how this should behave. We should think through the following:
  • What is the best gesture to disable autoupdate? (Useful to isolate repo, esp when the rapid churn of CLI updates break stuff).
  • Should autoupdate be opt-in instead of on by default?
  • Should updates to korebuild invalidate previous builds? e.g. forcing a clean and rebuild.
  • How can devs check which version of KoreBuild is in use?
  • How can a dev build with an old version of KoreBuild? Scenario: I want my build to run like it did yesterday for (insert reason here.)

[natemcmaster]

PR to update Universe to use the compiled KoreBuild - https://github.com/aspnet/Universe/pull/521