search

volcano-sh / volcano

A Cloud Native Batch System (Project under CNCF)
https://volcano.sh
Apache License 2.0
4.23k stars 969 forks source link

feature:add preemptionpolicy in preempt and reclaim #3739

Closed JesseStutler closed 3 weeks ago

JesseStutler commented 1 month ago

ref: #3642 ,#3738

JesseStutler commented 1 month ago

/assign @hwdef @Monokaix @lowang-bh Please take a look~

JesseStutler commented 1 month ago

Verification

test-1

Create two jobs, job-a and job-b, where the PriorityClass specified by job-b has a higher priority than the PriorityClass specified by job-a, but the PriorityClass specified by job-b comes with PreemptionPolicy: Never.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 200
preemptionPolicy: Never

The Pod created by job-b does not specify a PriorityClassName, so the PriorityClass specified by job is used.： 1727161142527

The Pod created by job-b fails to preempt because PreemptionPolicy is set to true:： image

test-2

Create two jobs, job-a and job-c, job-c has a specified PriorityClass without the PreemptionPolicy field, but the task has a different PriorityClass with the PreemptionPolicy field specified, and job-a remains unchanged.

job-c yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 200
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: cover
value: 200
preemptionPolicy: Never
---
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: job-c
spec:
  schedulerName: volcano
  minAvailable: 4
  priorityClassName: high-priority
  tasks:
    - replicas: 4
      name: "test"
      template:
        spec:
          priorityClassName: cover
          containers:
            - image: alpine
              command: ["/bin/sh", "-c", "sleep 6000"]
              imagePullPolicy: IfNotPresent
              name: running
              resources:
                requests:
                  cpu: 3900m
          restartPolicy: OnFailure

The PriorityClassName specified by job-c's pod is the task template's image Preemption failure log: image

test-3

Submit two jobs, one job-a and one job-b, in two separate queues, first submit job-a to fill up the cluster resources, then submit job-b, job-a needs to reclaim half of the resources to job-b, but because the PriorityClass specified by job-b specifies PreemptionPolicy as Never, job-a cannot reclaim job-a's resources, so job-b's pod all are pending. image

JesseStutler commented 1 month ago

/cc @Monokaix

hwdef commented 1 month ago

/lgtm

Monokaix commented 3 weeks ago

/ok-to-test

Monokaix commented 3 weeks ago

/lgtm /approve

volcano-sh-bot commented 3 weeks ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Monokaix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/volcano-sh/volcano/blob/master/OWNERS)~~ [Monokaix] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment