ci: refactor asdf plugin installations

[jhchabran]

When a fresh e2e agent is being deployed on GCP, it will fail if some plugins are not installed.

The current approach to address this is to install the plugins directly in the agent when baking the images and to then let each repository handle the installation of each tool version themselves.

If a plugin was not added and is present in .tools-version, calling asdf install will fail seen above.

There are multiple issues with our current approach:

• We are calling asdf install in multiple places:
  • https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/dev/ci/test/setup-deps.sh?subtree=true
  • https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/.buildkite/hooks/pre-command?subtree=true
  • https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/.buildkite/hooks/post-checkout?subtree=true
• Every time a plugin is added to a repository, we need to update the base agent images to add that plugin.
• Some plugins will introduce failures because they got updated (for example, see https://github.com/asdf-vm/asdf-nodejs/pull/214/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L29-R29) which will make the images fail.

➡️ As we are moving to stateless agents, it may make sense to let each repository handle grabbing its plugin on its own, merely providing agents with a bare asdf installed.

[jhchabran]

@bobheadxi What about, along the way, also getting rid of .nvmrc + LEGACY_TOOLS_VERSION=1 in .asdfrc? We have talked about it a few times but never did it. This task scope and is required to QA'it would cover it as well :)

That would simplify stuff in:

• sg setup
• CI scripts
• debugging :D (see https://github.com/sourcegraph/deploy-sourcegraph/pull/4054)

[jhchabran]

I just recalled that we also need to clean up https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/.buildkite/hooks/pre-command?L6-27 which, being run in a pre-command is rather heavy-handed. Moving it to a post-checkout out would be saner I think.

[bobheadxi]

Update: I've been struggling to get the build to run smoothly on my machine, and am going to revisit this next week in favour of some smaller tasks and SOC2 work. The script in https://github.com/sourcegraph/infrastructure/pull/3019 works locally though

Hopefully company laptop arrives sooner rather than later 😬

[bobheadxi]

Reopening - agents with the plugin installation has rolled out, I am going to keep this open to work through the refactors

[bobheadxi]

@jhchabran I'm going to close this with https://github.com/sourcegraph/sourcegraph/pull/30217 and https://github.com/sourcegraph/infrastructure/pull/3064. A bunch of other repos has this "kickstart" clutter that adds asdf installs and asdf plugin-add scripts - it seems to be used in various instructions and automation to generate GitHub actions using... dhall? I think cleaning up all this stuff is out of scope, and if it works now I'd rather not make changes to a whole bunch of repos that are not that critical. Most don't seem to run in Buildkite anyway

The rest of the "more important" deploy-sourcegraph repos all seem to only have asdf install in pre-command, which is good: https://sourcegraph.com/search?q=context%3A%40sourcegraph%2Fall%20r%3Adeploy-sourcegraph%20asdf%20install and no usages of plugin-add, which is also good: https://sourcegraph.com/search?q=context:%40sourcegraph/all+r:deploy-sourcegraph+asdf+plugin-add&patternType=literal