Ability to add kubernetes annotations to prefect KubernetesJobs

[Wirick]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar request and didn't find it.
• [X] I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Right now, there is no way for annotations to be added to a KubernetesJob manifest from my understanding. This sits similarly next to labels in the metadata block of the job manifest.

Describe the proposed behavior

I would like to be able to specify annotations in my job template manifest and have them populated to my kubernetes job (prefect flow)

Example Use

I need to put an annotation on my kubernetes job to tell an admission controller to populate secrets into the job environment

Additional context

Annotations are pretty straightforward to implement but aren't totally trivial. I implemented a quick fix that I would be happy to iterate on :) it's a little more subtle than labels because annotations aren't in the default JobManifest, so the json application of shortcuts errors out when it is trying to apply an annotation to a doc that doesn't exist. I made a little intermediate function to inject noop defaults into kubernetes job manifests, but I'm not sure if I'm just missing this functionality somewhere else so feel free to point me!

[serinamarie]

Hi @Wirick, welcome and thanks so much for reaching out and bringing this issue to our attention! We also appreciate your contribution to work to resolve this issue and will take a look at your pull request 🙂

[Wirick]

Thanks so much!

[rdenham]

This is something I'd like to see as well. In my manifest, I need to have something like:

apiVersion: v1
kind: Pod
metadata:
  name: examplepod
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
  namespace: prefectns

but while I can add the annotations section to the base job template, it doesn't populate the actual manifest. I'd even be ok modifying the python code to do it in a hacky way, but I got confused about where to put it.

We are running a self hosted server, and I submit a job from a separate client machine.

I thought it would be as simple as editing 'src/prefect/infrastructure/kubernetes.py' but my changes didn't seem to make a difference. Can anyone point to the simplest way of adding a custom annotations section to the metadata of the pod manifest?