What happened?: Getting HPA reporting status unknown for a given metric, other metrics seem to work fine:
"error_rate_metric" on Ingress/my-ingress (target value): <unknown> / 1
...
Warning FailedGetObjectMetric 83s (x95 over 25m) horizontal-pod-autoscaler unable to get metric error_rate_metric: Ingress on my-namespace my-ingress/unable to fetch metrics from custom metrics API: the server could not find the metric error_rate_metric for ingresses.networking.k8s.io my-ingress
What did you expect to happen?: Custom metric reports back with at least 1 given the query being used
Please provide the prometheus-adapter config:
The config for this metric is fairly simple, and in theory should always return SOME value via the clamp_min:
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric error_rate_metric: Ingress on my-namespace my-ingress/unable to fetch metrics from custom metrics API: the server could not find the metric error_rate_metric for ingresses.networking.k8s.io my-ingress
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:
Verbose logging in the adapter shows the following when trying to request the data from the HPA:
However when I attempt to query it I get a NotFound:
❯ kubectl --context=cluster-context get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/my-namespace/Ingress/my-ingress/error_rate_metric" | jq .
Error from server (NotFound): the server could not find the metric error_rate_metric for Ingress my-ingress
What happened?: Getting HPA reporting status
unknown
for a given metric, other metrics seem to work fine:What did you expect to happen?: Custom metric reports back with at least 1 given the query being used
Please provide the prometheus-adapter config:
The config for this metric is fairly simple, and in theory should always return SOME value via the
clamp_min
:Please provide the HPA resource used for autoscaling:
Please provide the HPA status:
Please provide the prometheus-adapter logs with -v=6 around the time the issue happened:
Verbose logging in the adapter shows the following when trying to request the data from the HPA:
Other logs were present but not relevant the this error rate metric failing
Anything else we need to know?:
Querying prometheus for the what I expect it should be translating to shows data in response, to be clear though, the results have no labels:
When querying via the RAW addresses in kubectl, I can see that this named metric does exist:
However when I attempt to query it I get a
NotFound
:I expect this to at the very least show 1.
It seems like this issue may be related to this, however the fixes in there do not seem to have helped: https://github.com/kubernetes-sigs/prometheus-adapter/issues/150
Environment:
prometheus-adapter-4.3.0
0.31.0
kubectl version
):