WIP: This currently doesn't work. The controller container simply doesn't have permissions to execute a script. (That sounds sensible actually?) So it will have to go in another container.
The stack can configure the agent to run hooks easily. In fact the agent container ships with a config file that sets the hook path to /buildkite/hooks, so one merely need mount some hooks at that path and the agent will run them, including pre-bootstrap.
However, with podSpecPatch, a pipeline can trivially override the hooks path (e.g. setting a BUILDKITE_HOOKS_PATH env var) which would disable pre-bootstrap being run. So pre-bootstrap no longer works as a mechanism to check jobs before running.
This PR adds a new special hook for the k8s controller: pre-schedule. Similar to pre-bootstrap it has the ability to inspect the job the controller is about to schedule a pod for. Rather than "env var" form, I've opted to dump the whole job as JSON.
Since this happens in the scheduler, I've gone ahead with a long-desired refactor of some of the scheduler methods to eliminate the err struct field.
WIP: This currently doesn't work. The controller container simply doesn't have permissions to execute a script. (That sounds sensible actually?) So it will have to go in another container.
The stack can configure the agent to run hooks easily. In fact the agent container ships with a config file that sets the hook path to
/buildkite/hooks, so one merely need mount some hooks at that path and the agent will run them, includingpre-bootstrap.However, with
podSpecPatch, a pipeline can trivially override the hooks path (e.g. setting aBUILDKITE_HOOKS_PATHenv var) which would disablepre-bootstrapbeing run. Sopre-bootstrapno longer works as a mechanism to check jobs before running.This PR adds a new special hook for the k8s controller:
pre-schedule. Similar topre-bootstrapit has the ability to inspect the job the controller is about to schedule a pod for. Rather than "env var" form, I've opted to dump the whole job as JSON.Since this happens in the scheduler, I've gone ahead with a long-desired refactor of some of the scheduler methods to eliminate the
errstruct field.