[Bug] ray cluster service not selecting worker node

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

Hi,

I'm using kuberay to run a NodePort ray cluster service on a local k8s cluster and there is a ray serve application running on the ray cluster. Since ray starts a httpproxy on each node, I want the incoming requests to somehow equally assigned to each node and then route to a selected serve replica. However, I noticed that the raycluster service is only selecting the head pod, which means all throughput are first sent to the head pod's httpproxy and that lead to a overall low throughput of the service. I read the official documentation and know about the adding a ingress as the load balancer stuff. But I'm still wondering even without an ingress, k8s should be able to perform the load balance in a random or round-robin manner and ease the head node httpproxy bottleneck issue. Below is the config of the service, the last two selector filtered all worker nodes.

Name:                     rayjob-langid-raycluster-4xd9k-head-svc
Namespace:                ray
Labels:                   app.kubernetes.io/created-by=kuberay-operator
                          app.kubernetes.io/name=kuberay
                          ray.io/cluster=rayjob-langid-raycluster-4xd9k
                          ray.io/identifier=rayjob-langid-raycluster-4xd9k-head
                          ray.io/node-type=head
Annotations:              <none>
Selector:                 app.kubernetes.io/created-by=kuberay-operator,app.kubernetes.io/name=kuberay,ray.io/cluster=rayjob-langid-raycluster-4xd9k,ray.io/identifier=rayjob-langid-raycluster-4xd9k-head,ray.io/node-type=head           
# Only head node is selected

KubeRay version: 0.5.0 Ray version: 2.6.3

I'm a beginner to k8s, sorry if this is a stupid question and possibly it's not a bug. I'm also glad to know your thought on the designing of the selectors. Thank you!

Reproduction script

apiVersion: ray.io/v1alpha1
kind: RayJob
metadata:
  name: rayjob-langid
spec:
  entrypoint: <run script to start my serve application>
  # rayClusterSpec specifies the RayCluster instance to be created by the RayJob controller.
  rayClusterSpec:
    rayVersion: '2.6.3' # should match the Ray version in the image of the containers
    # Ray head pod template
    headGroupSpec:
      serviceType: NodePort # optional
      # The `rayStartParams` are used to configure the `ray start` command.
      # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
      # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
      rayStartParams:
        dashboard-host: '0.0.0.0'
      #pod template
      template:
        spec:
          imagePullSecrets:
            - name: image-pull-credentials
          containers:
            - name: ray-head
              image: <image>
              ports:
                - containerPort: 6379
                  name: gcs-server
                - containerPort: 8265 # Ray dashboard
                  name: dashboard
                - containerPort: 8000
                  name: serve
              resources:
                limits:
                  cpu: "2"
                  memory: 4Gi
                requests:
                  cpu: "2"
                  memory: 4Gi
              volumeMounts:
                - name: models
                  mountPath: /etc/models
          volumes:
          - name: models
            persistentVolumeClaim:
              claimName: models-prod
    workerGroupSpecs:
      # the pod replicas in this group typed worker
      - replicas: 2
        minReplicas: 2
        maxReplicas: 2
        # logical group name, for this called small-group, also can be functional
        groupName: small-group
        # The `rayStartParams` are used to configure the `ray start` command.
        # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
        # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
        rayStartParams: {}
        #pod template
        template:
          spec:
            containers:
              - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
                image: <image>
                lifecycle:
                  preStop:
                    exec:
                      command: [ "/bin/sh","-c","ray stop" ]
                resources:
                  limits:
                    cpu: "2"
                    memory: 4Gi
                  requests:
                    cpu: "2"
                    memory: 4Gi
                volumeMounts:
                - name: models
                  mountPath: /etc/models
            volumes:
            - name: models
              persistentVolumeClaim:
                claimName: models-prod

Anything else

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

ray-project / kuberay