"Pod fits on node" that has lower utilization than current node

[ArGJolan]

What version of descheduler are you using?

descheduler version: 0.30.1

Does this issue reproduce with the latest release?

Yes

Which descheduler CLI options are you using? --v7 --dry-run

Please provide a copy of your descheduler policy config file

apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
maxNoOfPodsToEvictPerNamespace: 1
profiles:
- name: PodPacking
  pluginConfig:
  - args:
      evictLocalStoragePods: true
      ignorePvcPods: true
      nodeFit: true
    name: DefaultEvictor
  - args:
      evictableNamespaces:
        exclude:
        - kube-system
      thresholds:
        cpu: 70
        memory: 70
    name: HighNodeUtilization
  plugins:
    balance:
      enabled:
      - HighNodeUtilization
strategies: null

What k8s version are you using (kubectl version)?

kubectl version Output

$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1-eks-49c6de4

What did you do? Configured the scheduler of the cluster to use MostAllocated Deployed descheduler with the above HighNodeUtilization policy

What did you expect to see? With the current policies, I expect pods on nodes < 70% memory / CPU usage to get descheduled if there is room on another node with higher usage.

What did you see instead? Some pods are sometimes descheduled when the only other node they can fit on has lower resource usage, resulting in an endless loop of descheduling the pods (and having them rescheduled on the same node since their usage is higher)

Here are the truncated logs:

I0709 12:23:53.318800       1 nodeutilization.go:204] "Node is underutilized" node="ip-x-x-x-47.us-east-1.compute.internal" usage={"cpu":"2520m","memory":"7576Mi","pods":"12"} usagePercentage={"cpu":64.29,"memory":52.14,"pods":20.69}
I0709 12:23:53.319168       1 nodeutilization.go:204] "Node is underutilized" node="ip-x-x-x-63.us-east-1.compute.internal" usage={"cpu":"220m","memory":"344Mi","pods":"6"} usagePercentage={"cpu":5.61,"memory":2.37,"pods":10.34}

[...]

I0709 12:23:53.329329       1 node.go:154] "Pod fits on node" pod="test/test-pod" node="ip-x-x-x-63.us-east-1.compute.internal"
I0709 12:23:53.329395       1 evictions.go:168] "Evicted pod in dry run mode" pod="test/test-pod" reason="" strategy="HighNodeUtilization" node="ip-x-x-x-47.us-east-1.compute.internal" profile="PodPacking"

We can see that the pod is currently scheduled on node ip-x-x-x-47.us-east-1.compute.internal with a utilization of {"cpu":64.29,"memory":52.14,"pods":20.69} and the descheduler considers that it can fit on ip-x-x-x-63.us-east-1.compute.internal with a (lower) utilization of {"cpu":5.61,"memory":2.37,"pods":10.34}

The pod is then descheduled, and because the scheduler is configured with the MostAllocated option, the pod gets scheduled on ip-x-x-x-47.us-east-1.compute.internal again.

[jolet]

experiencing same issue (k8s v1.29.3, descheduler version: 0.30.1) pod with node affinity that can land him on 3 nodes:

• nodeA(underUtilized, usagePercentage={"cpu":6,"memory":7.41,"pods":7.27})
• nodeB(underUtilized, usagePercentage={"cpu":10.38,"memory":9.51,"pods":9.09})
• nodeC(highly utilized, usagePercentage={"cpu":90,"memory":99,"pods":90})

using HighNodeUtilization with mostAllocated scoring strategy, pod scheduled on nodeB. When descheduler runs it decides to sweep the pod anyway which ends in endless descheduling loop. Expected if pod landed on nodeB it should stay there since nodeC is already highly utilized and nodeA is lower in utilization than current nodeB where pod resides

    apiVersion: "descheduler/v1alpha2"
    kind: "DeschedulerPolicy"
    maxNoOfPodsToEvictPerNamespace: 1
    profiles:
    - name: default
      pluginConfig:
      - args:
          nodeFit: true
          evictFailedBarePods: true
          evictLocalStoragePods: true
        name: DefaultEvictor
      - args:
          evictableNamespaces:
            exclude:
            - kube-system
          thresholds:
            cpu: 70
            memory: 90
        name: HighNodeUtilization
      plugins:
        balance:
          enabled:
          - HighNodeUtilization

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale