Hooks don't work when checkout is skipped

[bpoland]

Hey, I noticed that when we try disabling checkout (https://github.com/buildkite/agent-stack-k8s/pull/303) that our environment hooks don't run (the pre-checkout hook also doesn't run but I'd expect that :) )

Maybe the environment hooks normally get run on the checkout container which is no longer present when checkout is disabled?

potentially related issues:

• https://github.com/buildkite/agent-stack-k8s/issues/227
• https://github.com/buildkite/agent-stack-k8s/issues/239
• https://github.com/buildkite/agent-stack-k8s/pull/258

[DrJosh9000]

Hi @bpoland, that's kind of weird! It was working before?

I had a quick look in the agent and couldn't see a reason the environment hook wouldn't still be run in the command container(s). It's part of setUp which isn't conditional on any phases.

[bpoland]

Hi @bpoland, that's kind of weird! It was working before?

I had a quick look in the agent and couldn't see a reason the environment hook wouldn't still be run in the command container(s). It's part of setUp which isn't conditional on any phases.

Hey, yep I tried removing the kubernetes.skip.checkout: true and then the hooks run again, although I think likely on the checkout container? (hard to tell from the Buildkite UI which container it is).

Is the environment hook supposed to run on both the checkout container and the command container separately? With checkout enabled I only see it running once.

[DrJosh9000]

Yeah, as far as I can tell if both containers are enabled, and the environment hook is present in both, and the hooks env var is configured, then the environment hook should still get run twice.

I'll investigate to see if I can replicate

[DrJosh9000]

I couldn't replicate, but I'll paste my notes here and we can compare 🧑🏻‍💻

As setup, I have a directory of trivial testhooks containing most job lifecycle hooks except checkout and command. For testing purposes I'm mounting them using hostPath.

Here's a pipeline which skips checkout:

agents:
  queue: kubernetes
steps:
  - label: "hook time"
    command:
      - echo $$BUILDKITE_HOOKS_PATH
      - ls /buildkite/hooks
    plugins:
      - kubernetes:
          checkout:
            skip: true
          podSpecPatch:
            containers:
              - name: "container-0"
                image: ghcr.io/buildkite/agent:latest
                env:
                  - name: BUILDKITE_HOOKS_PATH
                    value: /buildkite/hooks
            volumes:
              - name: agent-hooks
                hostPath:
                  path: /Users/josh/testhooks
                defaultMode: 0755
          extraVolumeMounts:
            - name: agent-hooks
              mountPath: /buildkite/hooks
              readOnly: true

It ran the environment, pre-command, post-command, and pre-exit hooks:

Running global environment hook
Running global pre-command hook
Running commands
$ echo $BUILDKITE_HOOKS_PATH
ls /buildkite/hooks
/buildkite/hooks
environment    post-checkout  pre-artifact   pre-checkout   pre-exit
post-artifact  post-command   pre-bootstrap  pre-command
Running global post-command hook
Running global pre-exit hook

I ran the same pipeline with checkout: skip: true commented out and another container podSpecPatch for "checkout":

...
      - kubernetes:
          # checkout:
          #   skip: true
          podSpecPatch:
            containers:
...
              - name: "checkout"
                image: ghcr.io/buildkite/agent:latest
                env:
                  - name: BUILDKITE_HOOKS_PATH
                    value: /buildkite/hooks
...

and it ran environment, pre-checkout, post-checkout, environment (again), pre-command, post-command, and pre-exit hooks:

Running global environment hook
Running global pre-checkout hook
Preparing working directory
Running global post-checkout hook
Running global environment hook
Running global pre-command hook
Running commands
$ echo $BUILDKITE_HOOKS_PATH
ls /buildkite/hooks
/buildkite/hooks
environment    post-checkout  pre-artifact   pre-checkout   pre-exit
post-artifact  post-command   pre-bootstrap  pre-command
Running global post-command hook
Running global pre-exit hook

[bpoland]

ahhh 🤦 it looks like my hooks volume is not getting correctly mounted on the command container. I will figure out why that is but that's probably it. I assume then I'll see the env hook running twice (or just once if I disable checkout again). Sorry for the wild goose chase!