hpa get cpu utilization from deleted pod

zjj2wry commented 4 years ago

What happened:

wlx-prd              33m         Warning   FailedComputeMetricsReplicas   HorizontalPodAutoscaler   horizontal-pod-autoscaler            failed to get cpu utilization: did not receive metrics for any ready pods                                                                                                                 63d          30      epnu-dataquality.15cd6629a893ce13
$ kubectl get po epnu-dataquality.15cd6629a893ce13 -n wlx-prd
Error from server (NotFound): pods "epnu-dataquality.15cd6629a893ce13" not found

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"4304b26", GitTreeState:"", BuildDate:"2019-04-08T08:50:29Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-04-22T11:34:20Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Network plugin and version (if this is a network-related bug):
Others:

zjj2wry commented 4 years ago

/sig autoscaling

wy100101 commented 4 years ago

I'm running into this and it basically makes the HPA unusable for me because CPU spikes up during graceful termination due to cleanup activities for our app.

I've looked at the code as far I can understand the workings of things like kubernetes delete operations and it looks like a deleted pod should get skipped since DeletionTimestamp should be set before the TERM signal is set so the pods should be skipped by replica_calculator#GroupPods()

Is there a race condition or a bug in the code?

wy100101 commented 4 years ago

Is the issue that the HPA is using a caching lister to get the pods? Maybe the calculation is based on a cached view of the pod and the HPA doesn't realize the pod has been deleted.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

wy100101 commented 4 years ago

I think this is still an issue.

On Tue, Jun 9, 2020, 22:14 Jiajin Zheng notifications@github.com wrote:

Closed #86297 https://github.com/kubernetes/kubernetes/issues/86297.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/86297#event-3427505700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAG4C35MVFFCGCV2MMUWPDRV3T6PANCNFSM4J3FG63Q .

kubernetes / kubernetes

hpa get cpu utilization from deleted pod #86297