Open jrauschenbusch opened 10 months ago
Hello!
In addition, if you use ScaledObjects exclusively with CPU/memory triggers, the following metrics are not recorded at all:
- keda_scaler_active
- keda_scaler_errors
- keda_scaler_metrics_value
- keda_scaler_metrics_latency
You are right but there isn't any solution for that because CPU/Memory scalers are a wrapper over the k8s metrics server, so KEDA doesn't have those values because KEDA doesn't process them, it's the k8s metrics server who does it.
Honestly, I think that KEDA shouldn't take the control over those resources because the metrics.k8s.io api is reserved for k8s metrics server and it exposes some metrics like container or resource, and KEDA doesn't support them. I mean, KEDA can't know the cpu metric value if KEDA doesn't expose it, but IMHO KEDA shouldn't expose it instead of the current k8s metrics server. WDYT @zroubalik @tomkerkhove ?
All metrics are initialized after the bootstrapping phase with a default value so that they are rendered when requesting the /metrics endpoint so that Grafana dashboard variables can be evaluated correctly****
I think that we could do it, but there are some metrics attached to external resources for labeling, such as scaledobeject or scaledjobs, IDK if we can initialize them empty, but definitively KEDA should try to do something with them. Are you willing to contribute with this improvement?
You are right but there isn't any solution for that because CPU/Memory scalers are a wrapper over the k8s metrics server, so KEDA doesn't have those values because KEDA doesn't process them, it's the k8s metrics server who does it.
Go it. But imho at least the dashboard should still work out of the box. Maybe by evaluating the metric keda_scaled_object_errors
instead of keda_scaler_active
. Or there should be a suggestion to use a plain HPA-metrics-based dashboard for cpu/memory/cron-based scalers only. Maybe this should be a note in the Integrate with Prometheus page. Additionally a note for this behavior should also be present. I mean that some metrics will not appear if there are no external metrics used.
I think that we could do it, but there are some metrics attached to external resources for labeling, such as scaledobeject or scaledjobs, IDK if we can initialize them empty, but definitively KEDA should try to do something with them. Are you willing to contribute with this improvement?
You mean that for those missing metrics also the labels must be intialized? An alternative could be to just initialize the keda_scaler_active
metric within the skip-operation. It would mean that the dashboard would work out-of-the-box, but just a few charts will not be rendered as those metrics are not managed by KEDA.
Move https://github.com/kedacore/keda/blob/0386b17dade70970b476bc91ebb6e6aa15182a01/pkg/scaling/scale_handler.go#L603 below https://github.com/kedacore/keda/blob/0386b17dade70970b476bc91ebb6e6aa15182a01/pkg/scaling/scale_handler.go#L560
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
I'm also seeing this issue and would love to get some help.
I'm also seeing this issue and would love to get some help.
Use a plain Horizontal Pod Autoscaler (HPA) dashboard for CPU/Memory based scaled objects. Or do it like me and fix the dashboard by using HPA metrics instead of KEDA metrics. I removed some KEDA specific panels and modified the rest.
Can you share an example of such dashboard by any chance please?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed due to inactivity.
This issue has been automatically closed due to inactivity.
Report
The provided Grafana Dashboard does not work because the required metric
keda_scaler_active
is not rendered on the /metrics endpoint. Therefore, the dashboard variables for the dropdown boxes cannot be evaluated correctly. The reason is that not all metrics are correctly initialized at bootstrapping time (first recording with default values). In addition, if you use ScaledObjects exclusively with CPU/memory triggers, the following metrics are not recorded at all:keda_scaler_active
keda_scaler_errors
keda_scaler_metrics_value
keda_scaler_metrics_latency
Expected Behavior
keda_scaler_active
keda_scaler_errors
keda_scaler_metrics_value
keda_scaler_metrics_latency
Actual Behavior
Metrics are missing if one is just using ScaledObjects with CPU/Memory triggers only
Steps to Reproduce the Problem
Logs from KEDA operator
No response
KEDA Version
2.11.2
Kubernetes Version
1.25
Platform
Microsoft Azure
Scaler Details
No response
Anything else?
No response