Triggers for same type not working

[samuelteixeiras]

Report

I have a trigger configuration that uses 3 triggers from the same type prometheus, and the scale is not working for the first trigger metadata related to response time. triggers:

• type: prometheus metadata: [metadata1 - related to response time]
• type: prometheus metadata: [metadata2 - related to CPU metrics]
• type: prometheus metadata: [metadata3 - related to Memory Metric]

Expected Behavior

The scaler would scale up and down if the threshold of each Prometheus metric is reached.

Actual Behavior

The scaler is ignoring the response time metrics. My threshold is 20ms and my application is over that value and the number of pods is not increasing.

Steps to Reproduce the Problem

1. Set multiple triggers with the same type prometheus
2. Reach the different thresholds
3. Verify id the scaler is respecting all triggers

Logs from KEDA operator

example

KEDA Version

2.11.2

Kubernetes Version

1.28

Platform

Amazon Web Services

Scaler Details

ScaledObject

Anything else?

No response

[JorTurFer]

Hello, Are you using AverageValue or Value metric type for the time trigger? You haven't set it and as default KEDA uses AverageValue (as it's explained here), so if you have 20ms and 20 pods, the (average) value is 1, which is under the threshold indeed.

You can extend the scaling differences based on metric types here: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

[samuelteixeiras]

Hi, thank you for the reply. I did not set the type, so it's using the default the AverageValue. Saying that, all pods will have the same metric value: histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{namespace="my-app-namespace"}[5m])) by (le)) The value returned by the query is for all pods, if this returns 30ms, I believe it's 30ms for all k8s pods. So using AverageValue or Value would not make a difference.

Before I had only one Prometheus trigger and the Scaler was respecting the threshold

[JorTurFer]

Saying that, all pods will have the same metric value: histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{namespace="my-app-namespace"}[5m])) by (le)) The value returned by the query is for all pods, if this returns 30ms, I believe it's 30ms for all k8s pods. So using AverageValue or Value would not make a difference.

It makes the difference totally, as it's the HPA Controller who applies the difference 😄

You are returning 30ms with can be for all pod in your prometheus server, but using AverageValue the HPA controller will take that value and make the average using current pod count.

• AverageValue: if you have 30 pods, current value for the HPA controller will be 1, not 30 (because 30/30 => 1)
• Value: If you have 30 pods, current value for the HPA controller will still be 30

You can read more about the HPA algorithm here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details

Before I had only one Prometheus trigger and the Scaler was respecting the threshold

Was your original single trigger based on the response time? Your other triggers are probably okey, the only one that needs to be value 100% is the time based

[samuelteixeiras]

thank you once more:D , I will try to change it to Value, if works as expected I am going to close the issue.

Was your original single trigger based on the response time? Yes

[JorTurFer]

I will try to change it to Value, if works as expected I am going to close the issue.

Let me know how it goes :)

Was your original single trigger based on the response time? Yes

That's truly weird xD

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.