Feature Request: Ability to add arbitrary labels to the pods that get spun up to execute jobs

[Scalahansolo]

I have the build kite agent running in my cluster that is managed by ArgoCD. It doesn't seem like there is any way currently to have the actual jobs that get spun up by the agent have their labels altered? If I was able to add labels to those pods, I would be able to go ArgoCD resource tracking working properly.

[DrJosh9000]

Thanks for raising this @Scalahansolo!

You should be able to set annotations and labels with metadata within the kubernetes plugin. Something like (I'm not sure how ArgoCD operates):

steps:
  - label: Has labels
    plugins:
      - kubernetes:
          metadata:
            labels:
              argocd.argoproj.io/tracking-id: your-id-here

There's currently no config available to do this by default for all pods, would that be useful?

[Scalahansolo]

Ahh okay. I could do that!

Some extended thoughts there. Ideally anything I would be able to define as apart of steps via the K8s plugin, I could also do via the agent config. The purpose being that over here we have two "dev types" that would be interacting with BuildKite.

1. Infra / Devops engineers who aren't overly concerned about what's happening in the CI jobs themselves, but there are a handful of things they need to setup to make sure these runners are in a good state and can be managed and observed well.

2. The engineers who actually end up writing the CI jobs. These engineers typically have no idea how K8s works and the types of things we'd be defining in this plugin.

So yeah, either being able to define things like these labels, volume mounts, etc would be great at the agent level. Or... if there was a reusable piece we could add to this plugins section so our devs writing the jobs don't need to concern themselves with those details. Something like...

steps:
  - label: Has labels
    plugins:
      - motion_kubernetes_plugin

In this example, I could define a whole set of K8s related plugin information somewhere that just get reused everywhere this is used.

[DrJosh9000]

YAML anchors and aliases can help with that, e.g.

x-anchors:
  motion_kubernetes_plugin: &motion_kubernetes_plugin
    kubernetes: 
      metadata: ...

steps:
  - label: ...
    plugins:
      - *motion_kubernetes_plugin

There are working examples in the pipeline.yml for this repo. (for env vars and volume mounts in our case). But it is somewhat limited in that there's no way to "override" parts of a mapping pulled in using an alias. A configurable default at the agent level would be able to let the plugin config add to or override the defaults.