`cluster_autoscaler_unschedulable_pods_count` emits 1 when it should be 0

Which component are you using?: cluster-autoscaler

What version of the component are you using?: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.1

What k8s version are you using (kubectl version)?: expand the toggle below to see

kubectl version Output

$ kubectl version --output=yaml
clientVersion:
  buildDate: "2023-06-14T09:53:42Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3
  goVersion: go1.20.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2024-04-30T23:54:04Z"
  compiler: gc
  gitCommit: f75443c988661ca0a6dfa0dc01ea82dd42d31278
  gitTreeState: clean
  gitVersion: v1.28.9-eks-036c24b
  goVersion: go1.21.9
  major: "1"
  minor: 28+
  platform: linux/amd64

What environment is this in?: EKS

What did you expect to happen?:

sum(cluster_autoscaler_unschedulable_pods_count{type="unschedulable"}) to emit 0 when there are no unscheduable pods.

What happened instead?:

sum(cluster_autoscaler_unschedulable_pods_count{type="unschedulable"}) emits 1.

In my logs, I see: I0726 14:56:29.952248 1 static_autoscaler.go:565] No unschedulable pods

How to reproduce it (as minimally and precisely as possible):

I am not sure. Perhaps enter a scenario where 1 pod is unscheduable, and then scale up a node to have it schedule, to assert whether the metric decrements?

Anything else we need to know?:

We've waited a few days, but there has been no change. I also confirmed that there are no Pending pods in the cluster.

kubernetes / autoscaler

`cluster_autoscaler_unschedulable_pods_count` emits 1 when it should be 0 #7091