Fix issue with buildkite-agent Job API when forwarding the job to the VM

What / TL;DR

Fixes an issue with the latest version of buildkite-agent used in the xcode-15.3 VM image (and later ones) that prevents jobs from being transferred from the host to the VM

Why / Issue details

In the latest versions of buildkite-agent, the Job API experiment has been de-experimented and enabled by default.

As a result, buildkite-agent bootstrap now tries to create a Unix socket at the BUILDKITE_SOCKETS_PATH path, then exposes the created socket path and token as BUILDKITE_AGENT_JOB_API_SOCKET and BUILDKITE_AGENT_JOB_API_TOKEN env vars.

The issue is that the default value for this path (aka --sockets-path option of buildkite-agent bootstrap) is $HOME/.buildkite-agent/sockets, so when our hostmgr generate buildkite-job command generates the script to handle the job in the VM, it exports all BUILDKITE_* env vars in that script… including the BUILDKITE_SOCKETS_PATH which was resolved to /Users/administrator/.buildkite-agent/sockets on the host. This resulted in buildkite-agent bootstrap failing on creating socket directory: mkdir /Users/administrator: permission denied error.

How

Override the BUILDKITE_SOCKETS_PATH env var in the generated script to /opt/ci/var/tmp/sockets
Disallow forwarding BUILDKITE_AGENT_JOB_API_SOCKET and BUILDKITE_AGENT_JOB_API_TOKEN in the generated script

I also took the occasion of this PR to:

Remove obsolete addEnvironmentVariable calls for BUILDKITE_BUILD_CHECKOUT_PATH, BUILDKITE_HOOKS_PATH and BUILDKITE_PLUGINS_PATH pointing to /usr/local/var/…, as those are legacy paths; besides those env keys are part of either disallowedKeys or overriddenKeys, so were removed or overridden later in the code, before scriptBuilder.build() is called… so those particular addEnvironmentVariable calls were not impacting the generated script code after all .
Remove duplicate Paths.tempFilePath constant which had the exact same value as Paths.tempDirectory, and replace its only call site
Turn var to let in Paths constants and rearrange their order and grouping a bit
Add a trailing newline to generated script (to avoid odd stdout when debugging its content with cat, for example 😉 )

Testing

As it wasn't easy to test this without releasing and deploying a new hostmgr version to our Mac hosts, instead I:

SSH'd into one of the hosts (I picked MV-MKE-ARM64-014)

Manually modify /opt/ci/hooks/command script like below, to unset BUILDKITE_AGENT_JOB_API_SOCKET and BUILDKITE_AGENT_JOB_API_TOKEN and set BUILDKITE_SOCKETS_PATH to hardcoded value, and thus simulate the same change made in this hostmgr code:

administrator@MV-MKE-ARM64-014 hooks % diff -u /opt/ci/hooks/command{.bak,}             
--- /opt/ci/hooks/command.bak   2024-03-08 13:21:28
+++ /opt/ci/hooks/command   2024-03-08 13:23:59
@@ -35,7 +35,15 @@
echo "~~~ ➡️ Transferring Job to VM"
echo "Received Job: $BUILDKITE_JOB_ID"

-SCRIPT_PATH=$(hostmgr generate buildkite-job)
+echo ">>> Olivier's Testing"
+echo "env before:"
+env
+SCRIPT_PATH=$(unset BUILDKITE_AGENT_JOB_API_SOCKET; unset BUILDKITE_AGENT_JOB_API_TOKEN; export BUILDKITE_SOCKETS_PATH=/opt/ci/var/tmp/sockets; hostmgr generate buildkite-job)
+echo "=== Generated script ==="
+cat "$SCRIPT_PATH"
+echo
+echo "======"
+echo "<<< Olivier's Testing"

echo "Running $SCRIPT_PATH on builder@$IP_ADDRESS"
ssh builder@"$IP_ADDRESS" < "$SCRIPT_PATH"

Modified the DayOne-Apple pipeline in the pending Xcode-15.3 update PR, to enforce the job to run on that specific MV-MKE-ARM64-014 host
Validated that the Job successfully passed the step related to Job API in the VM and the transfer and running of the job to the VM working as expected, fixing the issue.
I then removed my patch of /opt/ci/hooks/command to restore MV-MKE-ARM64-014 to its previous state.

What's Next

Once this lands, I'll generate a new release of hostmgr (probably a non-beta 0.50.0) and work on deploying it (but probably not today, as it's a Friday and thus submission + code freeze day for many apps, so not the best day to interrupt CI (or risk breaking it during failed deployment 😅 ).

Automattic / hostmgr