Closed Duri9292 closed 1 month ago
Hello,
Which is the problem exactly? The value using m
(1040334m)?
In K8s context 'm' means mili
and it's used when the value is a float number because k8s doesn't use float numbers. When you see 1040334m
it means 1040,334
. In the same way, jumps between 600
and 580334m
are quite normal because it's jumping from 600
and 580,334
Hello @JorTurFer thank you for your quick response. The issue is that once the number is in mili
scale the HPA is always scaling to maximum possible replica number. When the average value is non float number the scaler is decreasing the replicas or scaling as expected.
e.g. current average number: 1741 replica:1
current average number: 580334m replica:3 I configured the trigger value to 5000 and the scaler is always active which should not be in this case.
Are you scrapping prometheus metric generated by KEDA? I almost sure that you have a peak which justifies the scaling out, as you said, you're under the threshold. The only option for that behaviour without a peak is that you have changed the target value and the HPA controller is still during the scaling cooldown (300 after the last scaling out)
The thing is that we turned down Event Hub data ingestion for the last 24 hrs which means that we are getting 0 incoming messages. (we wanted to test scaling to 0) So there are no peaks. Even a value like 1741
does not make very sense but if it is calculating the average value for the last few days it can be relevant. I will be monitoring the behavior once we enable data ingestion again.
Below is a graph for incoming messages to Event Hub (past 48hrs)
Data granularity: 5 minutes
No no, it doesn't use the average value at all. KEDA uses the current value, so if it's 0 in the eventhub and you don't see 0 in KEDA, it can be a misconfiguration or a bug. Do you see any value different from 0? You can manually query the metric value and check what KEDA returns: https://keda.sh/docs/2.14/operate/metrics-server/#querying-metrics-exposed-by-keda-metrics-server
Hello @JorTurFer to answer your question "Do you see any value different from 0?" yes, even when even hub was turned off the HPA had always some number in meterics.
We enabled the event hub again and for some reason, we stopped getting float values, and scaling is working as expected. Or at least I did not catch any float number during my observation since there is no history of this value I cannot confirm. But it seems that once the float values stopped occurring the scaling is ok.
No no, it doesn't use the average value at all.
The documentation mentions that these are average values, we are using default. If that is not true than sorry I must missed it.
However, the values from metrics still do not match values from event hub metrics.
Event Hub (sum) for the past 30 min
Event Hub (avg) for past 24 hrs
The documentation mentions that these are average values, we are using default. If that is not true than sorry I must missed it.
mb, I understood that KEDA recovers the average value from the eventhub. You are right and k8s workload will be scaled based on the average value calculated using the instant eventhub value
We enabled the event hub again and for some reason, we stopped getting float values, and scaling is working as expected. Or at least I did not catch any float number during my observation since there is no history of this value I cannot confirm. But it seems that once the float values stopped occurring the scaling is ok.
Float values are correct and they can happen, if eventhub returns 7 and you have 4 pods, you'll have a float value in average
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed due to inactivity.
Report
I'm getting incorrect values from the external metric. Sometimes the external metrics provide the correct values but most of the time the current values are way off.
Metrics examples: Here you can see that averageValue is 1040334m which does not make sense and it will trigger the maximum possible scaling.
From time to time the averageValue is more accurate and it looks more realistic.
Here are the incoming messages metrics directly from Azure and as you can see we have usually an average of 100 incoming messages per minute.![image](https://github.com/kedacore/keda/assets/28104456/59cfea05-34a6-416f-8267-0afa6ca524ba)
Expected Behavior
The Average values should be more consistent and showing the real values.
Actual Behavior
The current values are jumping from 600 to 580334m while the real average incoming message are usually around 100. We are processing approximately 22 000 messages per day so the average value like 580334m does not make any sense.
Steps to Reproduce the Problem
azure-eventhub
triggerLogs from KEDA operator
KEDA Version
2.11.2
Kubernetes Version
1.28
Platform
Microsoft Azure
Scaler Details
azure-eventhub
Anything else?
No response