Volcano Torchx job scaling issue on GKE with

[obeyda]

We are trying to use Volcano to run Torchx jobs in a GKE cluster, but none of the jobs are triggering a scale up on our GPU nodepools.

If we manually scale up the targeted GPU nodepool the jobs get scheduled as desired, but we can't afford to keep the nodes up all the time so we want the volcano jobs to be able to trigger the automatic scale up.

The Defaut Queue

``` apiVersion: scheduling.volcano.sh/v1beta1 kind: Queue metadata: generation: 11 name: default status: allocated: cpu: '0' memory: '0' pending: 1 reservation: {} state: Open spec: capability: cpu: '500' memory: 600Gi nvidia.com/gpu: '40' guarantee: {} reclaimable: true weight: 1 ```

The volcano job

``` apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: myvcjob-jl7rjr9wqc35x namespace: myapp-2681 status: conditions: - lastTransitionTime: '2024-01-23T19:59:03Z' status: Pending - lastTransitionTime: '2024-01-23T20:00:56Z' status: Running - lastTransitionTime: '2024-01-23T20:27:44Z' status: Failed minAvailable: 1 runningDuration: 42m13.924661549s state: lastTransitionTime: '2024-01-23T20:27:44Z' phase: Failed version: 4 spec: maxRetry: 3 minAvailable: 1 plugins: env: [] svc: - '--publish-not-ready-addresses' queue: default schedulerName: volcano tasks: - maxRetry: 3 minAvailable: 1 name: myvcjob-0 replicas: 1 template: metadata: annotations: sidecar.istio.io/inject: 'false' labels: app.kubernetes.io/instance: myvcjob-jl7rjr9wqc35x app.kubernetes.io/managed-by: torchx.pytorch.org app.kubernetes.io/name: app beta.kubernetes.io/instance-type: g4-standard-48 cloud.google.com/gke-gpu: 'true' cloud.google.com/gke-spot: 'false' node.kubernetes.io/instance-type: g4-standard-48 nvidia.com/gpu: present provisioning-model: on-demand sku: L4-Quad torchx.pytorch.org/app-name: app torchx.pytorch.org/replica-id: '0' torchx.pytorch.org/role-index: '0' torchx.pytorch.org/role-name: app torchx.pytorch.org/version: 0.6.0 volcano.sh/gpu-memory: '40000' spec: affinity: {} containers: - command: - bash - '-c' - >- newrelic-admin run-program torchrun --rdzv_backend c10d --rdzv_endpoint localhost:0 --rdzv_id 'myvcjob-jl7rjr9wqc35x' --nnodes 1 --nproc_per_node 4 --tee 3 --role '' -m myapp.app.components.app --job_id f5b3d845-e111-49bf-92fd-c935e50fa0da env: - name: TORCHX_TRACKING_EXPERIMENT_NAME value: default-experiment - name: LOGLEVEL value: WARNING - name: TORCHX_JOB_ID value: kubernetes://torchx/myvcjob-jl7rjr9wqc35x - name: TORCHX_RANK0_HOST value: localhost image: >- myimage:mytag name: myvcjob-0 ports: - containerPort: 29500 name: c10d protocol: TCP resources: limits: cpu: '44' memory: 45G nvidia.com/gpu: '4' requests: cpu: 43900m memory: 43976M nvidia.com/gpu: '4' securityContext: {} volumeMounts: - mountPath: /dev/shm name: dshm restartPolicy: Never tolerations: - effect: NoSchedule key: sku operator: Equal value: L4-Quad - effect: NoSchedule key: cloud.google.com/gke-spot operator: Equal value: 'false' - effect: NoSchedule key: nvidia.com/gpu operator: Equal value: present volumes: - emptyDir: medium: Memory name: dshm ```

Scheduler Config

``` actions: "enqueue,allocate,reclaim,backfill" tiers: - plugins: - name: priority - name: gang enablePreemptable: false - name: conformance - plugins: - name: overcommit - name: drf enablePreemptable: false - name: predicates arguments: predicate.VGPUEnable: true - name: proportion - name: nodeorder - name: binpack ```

This is just hapennig with volcano jobs, any other pod that we create can trigger the scale up without issues

[Monokaix]

Hi, can you paste pod yaml output? I saw that job entered a failed state.

    - lastTransitionTime: '2024-01-23T20:27:44Z'
      status: Failed