[bug] post-checkout hooks are inoperable

[agates4]

Because there isn't a checkout step (it's actually a command step under the hood), any configured post checkout hooks won't execute because there isn't an official checkout

[moskyb]

hey @agates4! can you go into a bit more detail as to what this behaviour is preventing you from doing? are there specific bits that need to happen at post-checkout for you, or is there an existing plugin you're trying to use that uses this hook?

part of the k8s stack's plugin's role is to take control of the checkout and exec process, so it might not be feasible for post-checkout hooks to run, but we can take a look.

[agates4]

hey @agates4! can you go into a bit more detail as to what this behaviour is preventing you from doing? are there specific bits that need to happen at post-checkout for you, or is there an existing plugin you're trying to use that uses this hook?

part of the k8s stack's plugin's role is to take control of the checkout and exec process, so it might not be feasible for post-checkout hooks to run, but we can take a look.

Hey @moskyb I don't have a specific use case for this hook just wanted to point out what does not have parity with the longstanding Buildkite agent functionality!

[triarius]

@agates4 There's another issue for a more customiseable checkout, and I think this could fall under that. Are you happy for use to consolidate these?

I think if we were going to start work in this area, one option I'm favouring is to allow the user to specify their own container definition to do the checkout, and bypass the default checkout container.

That container could just run a buildkite plugin and use the hooks in that, or it could do something else entirely, but we should establish a contract along the lines of it should check out the repo to /workspace/$BUILDKITE_AGENT_NAME/$BUILDKITE_PIPELINE/ or something like that so that the job conatiners know where to find it.

[bpoland]

👋 hey I'm just getting started with this plugin and for me, none of the hooks were working by default but I was able to get fix it by setting the BUILDKITE_HOOKS_PATH and now I'm seeing all the expected hooks running.

I opened a PR to have this done by default, and just added a comment there on a workaround in the mean time :) https://github.com/buildkite/agent-stack-k8s/pull/258

[bpoland]

Just circling back here, I can get global post-checkout hooks to work using the method I linked above but it seems like plugin post-checkout hooks aren't working for me. Specifically, this plugin -- the environment hook gets called but not the post-checkout one to write secrets to a file.

Looking closer at what's being executed and in what order, it doesn't appear to line up with the hooks documentation.

Specifically:

• the global/agent environment hook gets run twice? Once at the start (as expected) and again after the post-checkout hook
• the non-vendored plugin environment hook doesn't get run until after checkout (looks like it isn't even downloaded until after checkout)
• the non-vendored plugin post-checkout hook doesn't run (possibly because of the same reason?)

It seems like agent-stack-k8s should run the hooks in the same order as the docs say (and, I assume, other agent types). At the very least it should run the post-checkout hook later and document the deviation in its own README (and preferably in the buildkite docs as well).

[moskyb]

hey! hopefully i can shed some light on differences between the way hooks work in the vanilla agent cf. the kubernetes stack

the global/agent environment hook gets run twice

this is because the k8s stack executes the checkout and the command execution in seperate containers within the pod - they both get launched, and because within the agent, the environment hook is unskippable, it always gets run by both containers. the k8s-stack controller does some magic to stitch the logs from the checkout container and the command container together when they get sent back to buildkite.

the non-vendored plugin environment hook doesn’t get run until after checkout

this occurs as a consequence of two factors: a) the split between checkout and command execution, and b) because in the checkout container, we disable plugins. when the checkout container runs, it skips over the step where it would check out and run plugins, but once the command execution container runs, it has plugins enabled, and so it picks up and runs the plugin env hook

the non-vendored plugin post-checkout hook doesn’t run

as you mention in your post, this is for the same reason - plugins are disabled in the checkout container.

in the short to medium term, i don’t think there’s much we can do about the order of hook firings being different between the k8s stack and the agent, because of the split container logic, but we can and should add some documentation.

wrt. plugin hooks not firing, i believe this can be fixed by changing this line to Value: "plugin,checkout", - this will make the checkout container check out and execute plugins hooks, though only for the environment and (pre-, post- ) checkout hooks.

[bpoland]

@moskyb thanks for the details! I definitely think this would be useful information to have in the agent-stack-k8s documentation (and potentially also a note in the hooks docs that links to it?)

So if we were to change the line you mentioned then that would run the post-checkout hook for the plugin I want to use but it sounds like that would be on the checkout container and not the command execution one. So I guess I'd have to make sure the secret files are in a volume that's shared between containers (e.g. workspace) right?

That seems like a reasonable workaround but is there a way to configure the line you mentioned without forking agent-stack-k8s?

[DrJosh9000]

Hi @bpoland, just to let you know we haven't forgotten you - we'll definitely make post-checkout hooks work, and have two paths in mind for doing that. Stay tuned!

[DrJosh9000]

I've just released v0.9.0 which, among other things, enables plugins in the checkout container (per @moskyb's comment). This should make pre- and post-checkout hooks provided by plugins work. Please try it and let us know how it goes!

[bpoland]

I've just released v0.9.0 which, among other things, enables plugins in the checkout container (per @moskyb's comment). This should make pre- and post-checkout hooks provided by plugins work. Please try it and let us know how it goes!

Cool, thanks -- will try it out!