Prefect-server HPA over-scalling issue on CPU

[MikaelFDA]

cluster: AKS kubernetes: 1.27.9 prefect-version: 2.19.2 chart version: prefect-server-2024.5.23194919

My values.yaml is configured as follow :

server:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 5
    targetCPU: 90
    targetMemory: 90

  resources:
    requests:
      cpu: 500m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 512Mi

When I look at the HPA, this is what I see :

> kubectl get hpa
NAME             REFERENCE                   TARGETS            MINPODS   MAXPODS   REPLICAS   
prefect-server   Deployment/prefect-server   127%/90%, 2%/90%   1         5         5         

> kubectl top pods
NAME                                         CPU(cores)   MEMORY(bytes)
prefect-server-c6ff8cc88-9zj75               8m           328Mi           
prefect-server-c6ff8cc88-fw8hw               12m          331Mi           
prefect-server-c6ff8cc88-gxlbl               12m          327Mi           
prefect-server-c6ff8cc88-ppzlr               11m          326Mi           
prefect-server-c6ff8cc88-sdwgq               11m          329Mi

I tried to find out why the autoscaller says it's 127% but I cant find teh reason. On the same pool I have prefect-worker with HPA who doesn't have this problem.

Do you have any idea why it's happening ? Is it a bug with the chart or something with my cluster ?

[parkedwards]

thanks for opening @MikaelFDA - would you mind sharing the rendered prefect-server deploy and HPA manifests?

[parkedwards]

hey @MikaelFDA - i just noticed the resource requests you've set for your prefect-server deployment. it looks like your memory requests are lower than you might want it

  resources:
    requests:
      memory: 256Mi
    limits:
      memory: 512Mi

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Utilization is the ratio between the current usage of resource to the requested resources of the pod.

another hint is that the ordering of your TARGETS in the HPA describe should be showing you the memory utilization / cpu utilization (in that order), bc that's how we define it in the HPA

https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/templates/hpa.yaml#L22-L38

in other words:

1. you've requested 256Mi of memory
2. your prefect-server pods are using ~328Mi
3. 328 / 256 = ~128% <-- this is the utilization you're seeing

I'd suggest requesting more memory your deployment - for ex., our default values set this at 512Mi

[MikaelFDA]

@parkedwards Indeed it's the cause of my issue. Didn't think about that because on my others services created with helm, the order is CPU/RAM :facepalm:.

I guess it's never too late to learn the intricacies of Kubernetes :laughing: Thank for helping !

However, on k8s docs (like here) it's the order CPU/RAM. If it's a standard, I think it would be good idea to update the chart

[parkedwards]

@MikaelFDA good call out. I will switch the order of them now, to avoid confusion 😄 . thanks again for flagging this!