ray-project / kuberay

A toolkit to run Ray applications on Kubernetes
Apache License 2.0
1.16k stars 373 forks source link

[Bug] [API Server] Can't specify cluster rayVersion in Ray Job #2109

Open han-steve opened 5 months ago

han-steve commented 5 months ago

Search before asking

KubeRay Component

apiserver

What happened + What you expected to happen

The version attribute for a Ray Job is currently not used. There is no way of specifying the Ray Cluster version when launching a Ray Job. While I'm not sure how this version number is used in the controller (didn't find anything after a quick search), it is preventing me from using job metadata: if !constraint.Check(v) { return "", fmt.Errorf("the Ray version must be at least 2.6.0 to use the metadata field") } We support version in the Cluster protobuf, but we don't have it in the ClusterSpec protobuf, which is used by the Job protobuf.

Reproduction script

Create a Ray Job with metadata will cause the deployment to fail:


curl -X 'POST' \
    '<url>/apis/v1/namespaces/<namespace>/jobs' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
    "name": "rayjob-test-long-running",
    "namespace": "<namespace>",
    "user": "3cp0",
    "version": "2.9.0", # ignored 
    "entrypoint": "python -c \"import time; time.sleep(100)\"",
    "metadata": {
      "description": "test pipeline"
    },
    "shutdown_after_job_finishes": true,
    "clusterSpec": {
      "headGroupSpec": {
        "computeTemplate": "default-template",
        "image": "rayproject/ray:2.9.0",
        "rayStartParams": {
          "dashboard-host": "0.0.0.0"
        },
        "labels": {
          "sidecar.istio.io/inject": "false"
        }
      },
      "workerGroupSpec": [
        {
          "groupName": "small-wg",
          "computeTemplate": "default-template",
          "image": "rayproject/ray:2.9.0",
          "replicas": 1,
          "minReplicas": 0,
          "maxReplicas": 1,
          "rayStartParams": {
            "metrics-export-port": "8080"
          },
          "labels": {
            "sidecar.istio.io/inject": "false"
          }
        }
      ]
    }
  }'```

### Anything else

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!