force a given version of gitlab-runner?

[umlaeute]

I've been using TART_EXECUTOR_INSTALL_GITLAB_RUNNER=true in order to have a gitlab-runner on my ghcr.io/cirruslabs/macos-monterey-xcode:latest-based jobs for uploading artifacts.

unfortunately, the gitlab-runner version installed via this method is rather old.

$ which gitlab-runner
/opt/homebrew/bin/gitlab-runner
$ gitlab-runner --version
Version:      15.3.0
Git revision: bbcb5aba
Git branch:   15-3-stable
GO version:   go1.17.13
Built:        2022-08-19T22:37:06+00:00
OS/Arch:      darwin/arm64

(Now homebrew currently ships gitlab-runner_16.6.1, so I figure the outdated version is because themacos-monterey-xcode image has been created in 2002, and homebrew uses the old sources (and i tend to activate things like HOMEBREW_NO_AUTO_UPDATE=1 on my CI systems, as i do not want to spend hours upgrading things like androind-ndk that I never ever need).

I also noticed, that gitlab-runner is either installed via brew (if available), or via curl: in the latter case, it will always download the latest version of the runner, whereas in the brew case it will install whatever is currently packaged.

I find that i probably do not want the latest and greatest gitlab-runner (nor some outdated version) on the VM, but one that matches the gitlab-runner that is running on the host controlling the VM.

So i wonder whether it would make sense to extend the TART_EXECUTOR_INSTALL_GITLAB_RUNNER to also take a specific version number for the executor, so i could use more values for this variable:

value	installs...
false	don't install
true	pick whatever version homebrew (if available) or curl find
brew	only install via brew, fail if not available
curl	only install via curl, fail if not available
latest	get the latest version of gitlab-runner (as published by gitlab)
${CI_RUNNER_VERSION}	the exact version of the runner that is used on the host (the CI_RUNNER_VERSION is a one of GitLab's predefined variables
1.2.3	this specific version

for the specific versions, tart would pick an appropriate installation method (though for simplicity's sake, i would just go for curl)

[cosmoshepcat]

I like this idea but were having an issue where no matter what the only method it uses to install gitlab-runner on the fly is via curl. Yesterday the upstream curl was having random issues and pipelines were failing to upload artifacts due to corrupted gitlab-runner inside the vm.

Reading the script in the repo here and even running it manually on one of our vms works but never works when gitlab runs via pipeline. Since curl is at /usr/bin in ventura/sonoma etc by default my only guess was the initial ssh to spin up the vm is somehow having a very limited PATH like /bin:/usr/bin

Sorry to hijack your issue!

[fkorotkov]

Should we just bundle some version of the runners within the VM like we do with GitHub Runner? If a VM will have slightly outdated binary, is it a problem?

[cosmoshepcat]

Should we just bundle some version of the runners within the VM like we do with GitHub Runner? If a VM will have slightly outdated binary, is it a problem?

I cant speak for the OP but that is what we are doing right now. Packer installs via brew inside the vm. We like the idea of the functionality to install during the tart prepare stage but so far it has never worked for our setup as detailed in #53

We also keep our gitlab-runner versions pretty close to current; the versions on the mac machines themselves get updated weekly. It is probably not a huge issue though if the version inside the vm is slightly older. We are generating new packer images monthly.

[fkorotkov]

@cosmoshepcat seems like a great approach. I guess we can also bundle it in our VMs so this branch will work:

https://github.com/cirruslabs/gitlab-tart-executor/blob/d41393f078dfc243d4752fcc99fea64630eaf7a0/internal/commands/prepare/install-gitlab-runner.sh#L7-L12

[cosmoshepcat]

@fkorotkov

Thats what I was referencing in my other issue #53 that code block does not work in our environment and not sure why. Even with both brew and gitlab-runner installed via packer inside the vm it never finds them and always uses curl to install.

[umlaeute]

Should we just bundle some version of the runners within the VM like we do with GitHub Runner? If a VM will have slightly outdated binary, is it a problem?

slightly outdated might not be a problem, but VM images tend to get stale.

and i did have bad experience in the past with VMs that were running more-than-slightly outdated versions of the gitlab-runner. iirc, i've also experienced problems with too new gitlab-runner binaries within the VMs (I cannot remember from the top of my head what went wrong exactly, but something along the lines of the VM-host running an gitlab-runner that was much older than the one within the VM; at least I do have code in my pre-tart deploy scripts that ensures that the gitlab-runner within the VM is exactly the same as the one on the host.

hence my original suggestions with being able to use the ${CI_RUNNER_VERSION} variable.

in this vein: how about adding an option to just copy the ´gitlab-runner` binary from the host into the VM

value	installs...
host	copies the gitlab-runner binary from the host into the VM

this obviously only works if the host and the VM are similar.