Prefect Worker Job Template not updating in the UI

[Montek30]

Hi guys, I am using the latest feature of using the prefect worker job template as configmap (https://github.com/PrefectHQ/prefect-helm/issues/334) but the job template is not reflecting in the prefect UI. The worker pods are rotating with new config but the UI still has the old configuration. Helmrelease prefect-worker, version "2024.6.4205506" :

      config:
        workPool: cnp-eu-dev-worker-pool
        limit: 20
        baseJobTemplate:
          existingConfigMapName: "cnp-eu-dev-worker" 

Configmap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cnp-eu-dev-worker
  namespace: data-platform-prefect
data:
  baseJobTemplate.json: |
    {}

Does anyone know if there's anything odd with my config.

[jamiezieziula]

Hi @Montek30 - if the workpool already exists, the base job template will be ignored.

[Montek30]

Oh, that means we will have to delete the previous worker pool and redeploy? If yes, are you planning to support updating the worker pool in your future releases? @jamiezieziula

[marcincuber]

@jamiezieziula could we please raise it as a feature request so that baseJobTemplate.json automatically updates the existing workerpool? Usecase: control the template updates through gitops pipeline.

Moreover, removing worker pool to deploy a new scheme is removing selected workpool from all prefect deployments which isn't right.

[desertaxle]

@marcincuber If you want to manage the base job template for your work pools using gitops, we recommend using the perfect CLI to update the base job template in your pipeline.

A command like this should work for your use case:

prefect work-pool update my-work-pool --base-job-template baseJobTemplate.json

[marcincuber]

@desertaxle we can do it with cli for sure. However, that defeats the purpose of having it as part of helm release where clearly we can set it (when worker pool doesn't exist) but not after.

[jamiezieziula]

Hi @marcincuber @Montek30 -- thanks for the feedback here. @desertaxle and I will be working over the next few weeks to support updating the work pool base job template to account for changes picked up the helm chart!

[parkedwards]

@marcincuber @Montek30 thank you for raising this - we just pushed an update to the chart to allow syncing the helm-configure base job template to the API. give it a whirl, and let us know what you think

https://github.com/PrefectHQ/prefect-helm/releases/tag/2024.7.15170357