Add CI pipeline for creating self-hosted runners

[dennisameling]

This is a CI pipeline to create one or multiple Azure arm64 VMs with GfW and the GitHub Actions runner preinstalled and provisioned. The VMs are deallocated immediately after creation so they are prepared for later use.

Here's an example run: https://github.com/dennisameling/azure-arm64-gh-actions-runner/actions/runs/3740740959/jobs/6349467898

The runner is then added to the repo correctly 🎉 it's offline because we deallocate the VM immediately after setup in the CI pipeline.

After running a job on the runner, it's automatically removed thanks to the --ephemeral flag:

[dscho]

Woooooow! Nice work @dennisameling!

Unfortunately, I am a bit too frazzled today to have a closer look, as I have been struggling with the Git for Windows v2.39.0(2) release. But I hope to get to it tomorrow, otherwise in the New Year (I am planning on leaving the computer turned off for quite a bit over the next two weeks).

This will be super awesome!

[dennisameling]

But I hope to get to it tomorrow, otherwise in the New Year (I am planning on leaving the computer turned off for quite a bit over the next two weeks).

That makes a ton of sense - thanks once again for all your quick and incredibly accurate reviews and feedback. It's a pleasure working on things together this way. I'm very excited that we're finally getting close to a sustainable solution for arm64 support after 2+ years, while (potentially) also paving the way for clang64 support at some point in the future 😄

Shall we focus our efforts on the git-extra stuff for now? Looks like that'll unblock quite some other things. You mentioned that the old references will need to be updated in quite some places. I'm thinking to create a PR with many small, easy-to-review commits to update those across the board. It'd be great if we can get this PR merged soon so that I can test with the actual packages from the GfW Pacman repos. WDYT?

[dscho]

You mentioned that the old references will need to be updated in quite some places. I'm thinking to create a PR with many small, easy-to-review commits to update those across the board.

Yes, that would be good. I think the best starting point will be git grep -w git-extra in build-extra (ignoring the versions/ hits) and probably also the wiki:

$ git -C /usr/src/build-extra grep -w git-extra -- ':(exclude)versions/'

I grouped and annotated output in the following. First, the parts that need to be changed when we switch from installing `git-extra` to installing `mingw-w64--git-extra` instead (which will also need that `replaces` addition to `PKGBUILD`): ``` commit-msys2.sh: sed -ne '/.* M\tvar\/lib\/pacman\/local\/git-extra-[1-9].*\/desc$/d' \ commit-msys2.sh: ':(exclude)var/lib/pacman/local/git-extra-*/desc' \ git-extra/update-via-pacman.bat:@ECHO Wrapping up: re-install git-extra git-extra/update-via-pacman.bat:@pacman -S --noconfirm git-extra git-extra/update-via-pacman.bat:@ECHO End of re-install of git-extra make-file-list.sh:git-extra openssh $UTIL_PACKAGES" please.sh: # git-extra rewrites some files owned by other packages, please.sh: 'pacman -S '$force' --noconfirm git-extra' || please.sh: die "Cannot update git-extra in %s\n" "$sdk" please.sh: # Make sure we have the git-extra package locally, as please.sh: 'pkg=/var/cache/pacman/pkg/$(pacman -Q git-extra | please.sh: pacman -Sw --noconfirm git-extra && please.sh: pacman -S --noconfirm git-extra && please.sh: pacman -S --noconfirm git-extra' || please.sh: 'pacman -S --noconfirm git-extra' && please.sh: 'pacman -S --noconfirm git-extra' please.sh: # if built-ins are still original hard-links, reinstall git-extra please.sh: 'pacman -S --noconfirm git-extra' please.sh: 'pacman -S --noconfirm git-extra' please.sh: (. /var/lib/pacman/local/git-extra-*/install && post_install) ``` These parts need to be augmented with the new package name: ``` get-sources.sh: git-extra|mingw-w64-x86_64-git|mingw-w64-i686-git|msys2-runtime|mingw-w64-x86_64-git-credential-manager|mingw-w64-i686-git-credential-manager|mingw-w64-x86_64-git-credential-manager-core|mingw-w64-i686-git-credential-manager-core|mingw-w64-i686-git-lfs|mingw-w64-x86_64-git-lfs|curl|mingw-w64-i686-curl|mingw-w64-x86_64-curl|mingw-w64-i686-wintoast|mingw-w64-x86_64-wintoast|bash|heimdal|perl|openssh) please.sh: git-extra) ``` Here are the parts that need to be changed once we rename the `git-extra/` directory to `mingw-w64-git-extra/` in `build-extra`: ``` .github/workflows/build-and-deploy.yml: if test git-extra != "$PACKAGE_TO_BUILD" && test "z${PACKAGE_TO_BUILD#mingw-w64-}" = "z$PACKAGE_TO_BUILD" .gitignore:/git-extra/pkg/ .gitignore:/git-extra/src/ .gitignore:/git-extra/git-extra-*.pkg.tar.xz .gitignore:/git-extra/git-extra-*.src.tar.gz git-extra/vimrc:" This source file comes from git-for-windows build-extra repository (git-extra/vimrc) please.sh:# 2) release a new Pacman package, e.g. git-extra or msys2-runtime please.sh:# ./please.sh build git-extra please.sh:# ./please.sh install git-extra please.sh:# ./please.sh upload git-extra ``` These lines won't need changing, at least not for a long time (until the time when we stop building the superseded `git-extra` package altogether): ``` git-extra/PKGBUILD:_realname="git-extra" git-extra/PKGBUILD:install='git-extra.install' git-extra/PKGBUILD:# The old package name, git-extra, was causing problems as it's _actually_ a MINGW-package. git-extra/PKGBUILD: conflicts=('mingw-w64-clang-aarch64-git-extra' 'mingw-w64-x86_64-git-extra' 'mingw-w64-i686-git-extra') git-extra/PKGBUILD:package_mingw-w64-x86_64-git-extra() { git-extra/PKGBUILD: conflicts=('git-extra' 'mingw-w64-clang-aarch64-git-extra' 'mingw-w64-i686-git-extra') git-extra/PKGBUILD: provides=('git-extra') git-extra/PKGBUILD:package_mingw-w64-i686-git-extra() { git-extra/PKGBUILD: conflicts=('git-extra' 'mingw-w64-x86_64-git-extra' 'mingw-w64-clang-aarch64-git-extra') git-extra/PKGBUILD: provides=('git-extra') git-extra/PKGBUILD:package_mingw-w64-clang-aarch64-git-extra() { git-extra/PKGBUILD: conflicts=('git-extra' 'mingw-w64-x86_64-git-extra' 'mingw-w64-i686-git-extra') git-extra/PKGBUILD: provides=('git-extra') git-extra/git-extra.install: # Revert change by prior versions of git-extra. git-extra/git-extra.install: # Revert change by prior versions of git-extra. git-extra/git-extra.install: sed -i -e '/^# Added by git-extra/{N;/^\(.*\n\)\?Ciphers +[a-z].*cbc/d}' /etc/ssh/ssh_config git-extra/git-extra.install: '# Added by git-extra' \ git-extra/git-extra.install: '# Added by git-extra' \ git-extra/git-extra.install.in: # Revert change by prior versions of git-extra. git-extra/git-extra.install.in: # Revert change by prior versions of git-extra. git-extra/git-extra.install.in: sed -i -e '/^# Added by git-extra/{N;/^\(.*\n\)\?Ciphers +[a-z].*cbc/d}' /etc/ssh/ssh_config git-extra/git-extra.install.in: '# Added by git-extra' \ git-extra/git-extra.install.in: '# Added by git-extra' \ git-extra/git-sdk.sh: printf "%s " git git-extra msys2-runtime installer \ git-extra/git-sdk.sh: git-extra|git-for-windows-keyring|mingw-w64-cv2pdb|\ git-extra/git-sdk.sh: pacman -Syyu git-extra && git-extra/git-sdk.sh: sdk $cmd git-extra && git-extra/git-sdk.sh: sdk init git-extra && git-extra/proxy-lookup.c: * included in Git for Windows' git-extra package. pacman-helper.sh: git-extra-*.pkg.tar.xz) arch=${file%.pkg.tar.xz}; arch=${arch##*-}; key=${arch}_mingw;; please.sh: git-extra) please.sh: git commit -s -m "git-extra: adjust checksums" PKGBUILD && please.sh: if test git-extra.install.in -nt git-extra.install please.sh: if ! git diff-files --quiet -- git-extra.install please.sh: "git-extra: regenerate .install file" \ please.sh: git-extra.install ``` There is also this line (please.sh:2063, not to be confused with please.sh:408) that also needs to be addressed when we stop building the superseded `git-extra` package: ``` please.sh: git-extra) ```

$ git -C git-for-windows.wiki/ grep -w git-extra

``` Install-inside-MSYS2-proper.md: pacman -S mingw-w64-x86_64-{git,git-doc-html,git-doc-man} git-extra Install-inside-MSYS2-proper.md:The `git-extra` package modifies the MSYS2 environment heavily (sometimes in ways that are tedious to undo). The package can be skipped if you want to keep your MSYS2 pristine. As of this writing, the package: Install-or-update-inside-MSYS2,-Cygwin-or-Git-for-windows-itself.md:With the script `getgit` introduced by [git-for-windows/build-extra#261](https://github.com/git-for-windows/build-extra/pull/261), you can download and run the script or just run `curl https://raw.githubusercontent.com/git-for-windows/build-extra/HEAD/git-extra/getgit | bash`. Making-an-installer.md:Short version: you will need to run `make strip` in `/usr/src/git` before `make install`, and you will also need to run to call `pacman -S git-extra` afterwards. Making-an-installer.md:## Why `pacman -S git-extra`? Making-an-installer.md:The `pacman -S git-extra` part is defined elsewhere yet: in `/usr/src/build-extra/please.sh`. Its purpose is to replace many hardlinked copies in `/mingw64/libexec/git-core/git-*.exe` with what Git for Windows calls the "Git wrapper": a small executable that does nothing else but call `git.exe` with the appropriate subcommand name. Package-management.md:- [build-extra](https://github.com/git-for-windows/build-extra) contains the `git-extra` package information, Technical-overview.md:In some cases, the change you want to make is not in git itself, but in the additional files needed to emulate the *ix environment git expects (things like `grep`, `find`, `cat`,...), or in additional helper files (e.g. `start-ssh-agent.cmd`). These files come from [Msys](https://github.com/git-for-windows/MSYS2-packages) and [Mingw](https://github.com/git-for-windows/MINGW-packages) packages. Please see the documentation for how to make [changes to these packages](Package-management#technical-details). Some files are also in the `build-extras` repo in the subdir [`git-extra`](https://github.com/git-for-windows/build-extra/tree/HEAD/git-extra) (the script which calls notepad as a commit message editor, diff filter for word files,...). The source code for these files are also under `/usr/src` but need to be checked out first (e.g. `cd /usr/src/MINGW-packages && git fetch && git checkout main`). Updating-your-SDK.md:1. Run `update-via-pacman.bat` in the command window (it will always re-install the `git-extra` package at the end) ``` I think we will need to change these in conjunction with switching over to installing `mingw-w64--git-extra` instead of `git-extra`.

[dscho]

Look at that!

Created by https://github.com/git-for-windows/git-for-windows-automation/actions/runs/3760042983... I'll push the changes I made on top of https://github.com/git-for-windows/git-for-windows-automation/pull/5/commits/6cfc1d0464eac917197d8f41ac4c91ec02097345 (and that I used to run this workflow) in a moment.

[dscho]

Created by https://github.com/git-for-windows/git-for-windows-automation/actions/runs/3760042983...

It looks as if the workflow basically sits on its hands for 7.5 minutes, waiting for Azure. I do not see anything in actions/arm-deploy that would allow us to "fire and forget" about this. I guess we can wait for the VM to be created, but it looks like a bit of a waste of build minutes to me.

Together with the fact that deallocating the VM takes another ~1¼ minutes, I wonder whether the strategy is worth the effort to always keep one runner pretty much ready to go at all times. We could simply spin up the runner on demand instead, after all, a ~8 minute queue time is not that terrible for jobs that will run only once in a while. But I guess it's nicer to wait only a minute for the VM to spin up and we're almost there...

@dennisameling what are your thoughts on this matter?

[dennisameling]

I guess we can wait for the VM to be created, but it looks like a bit of a waste of build minutes to me.

I agree it's a waste of build minutes, but we could leave it as-is for now and create an issue in https://github.com/Azure/arm-deploy/ for them to add a "fire and forget" feature. WDYT?

We could simply spin up the runner on demand instead, after all, a ~8 minute queue time is not that terrible for jobs that will run only once in a while

We could certainly start with that. Most builds/pipelines take 20-30+ mins anyway, so an additional 8 mins should be okay for now. We could then gradually improve speed/performance over time.

Overall, I don't really have a strong preference. Whatever works best for you in terms of speed/pricing.

[dscho]

we could leave it as-is for now and create an issue in https://github.com/Azure/arm-deploy/ for them to add a "fire and forget" feature. WDYT?

Let's leave it as-is for now and if it becomes a problem, think about using Azure CLI directly (no need to go through the arm-deploy Action).

We could simply spin up the runner on demand instead, after all, a ~8 minute queue time is not that terrible for jobs that will run only once in a while

We could certainly start with that. Most builds/pipelines take 20-30+ mins anyway, so an additional 8 mins should be okay for now. We could then gradually improve speed/performance over time.

Overall, I don't really have a strong preference. Whatever works best for you in terms of speed/pricing.

Let's continue with the pre-spun-up strategy.

[dscho]

Even if the build failed that used the self-hosted runner, the workflow run that created that self-hosted runner worked, so I merged it.