search

kubernetes-sigs / descheduler

Descheduler for Kubernetes
https://sigs.k8s.io/descheduler
Apache License 2.0
4.44k stars 664 forks source link

The getTargetNodes in RemoveDuplicates does not respect node resources and can mistakenly evict pods #1237

Closed bingzheliu closed 6 months ago

bingzheliu commented 1 year ago

descheduler version

0.27.1

k8s version

v1.28.0

Problem

The RemoveDuplicate plugin is supposed to get the feasible nodes that can place the duplicatePods before calculating which pods to delete. In particular, this line of code.

However, it currently only checks on the tolerance/taint and nodeSelector (code here). The node can be infeasible to the pod due to other reasons, i.e., the node does not have enough resources for more pods.

This issue can cause more pods to be mistakenly evicted if nodes can't hold the pods due to limited resources. And unending cycle of scheduling and eviction can occur.

What is expected

The getTargetNodes should consider the node resource fits and other scheduling restrictions of the pods (like NodePorts).

lowang-bh commented 1 year ago

/assign

a7i commented 1 year ago

@bingzheliu nodeFit by default is false so you may want to enable that.

If you are using descheduler/v1alpha2, the docs can be found here to configure the DefaultEvictor.

profiles:
  - name: ProfileName
    pluginConfig:
    - name: "DefaultEvictor"
      args:
        nodeFit: true

If you are using descheduler/v1alpha, the docs can be found here

  RemoveDuplicates:
     enabled: true
     params:
       removeDuplicates:
         nodeFit: true
a7i commented 1 year ago

However nodePorts is not one of our nodeFit checks. Would you be open to proposing a design for that?

bingzheliu commented 1 year ago

Thanks for the reply @a7i ! Sure, I can create a new issue for the nodePorts later.

Regarding the resource fit, I understand there is the nodeFit in the evictor. However, not implementing the node resource fit in this getTargetNodes function inside the RemoveDuplicates plugins can still cause unexpected issues.

Because the result of targetNodes is used to calculate the upperAvg that determines which pods to evict. upperAvg := int(math.Ceil(float64(ownerKeyOccurence[ownerKey]) / float64(len(targetNodes)))) code here.

This example shows how the issue can occur: s1 In this cluster, node 1 can only schedule one more pod due to being occupied by other deployments; and node 1/2 has less weight than node 3 for nodeAffinity. Hence, the scheduler decides to schedule the 5 pods as shown in the above graph.

If the getTargetNodes consider the resource, targetNodes = 2and upperAvg = ceil(5/2) = 3; then none of the pods should be evicted.

However, as now the getTargetNodes only considered taint and nodeSelector, the targetNodes = 3and upperAvg = ceil(5/3) = 2, causing one pod to be evicted on node 3 and scheduled again, and an ending eviction/scheduling cycle perpetuated.

Even setting the NodeFit in the eviction plugin does not help, as there is another node (node 2) in this case that can host the pod.

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 6 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/descheduler/issues/1237#issuecomment-2024199561): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.