[Hostnetwork] How should I set different ports for ray node pods?

surenyufuz commented 10 months ago

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

I have deployed a ray job with host network mode.

To avoid port conflicts between ray node pods on a kubernetes node, I should set different ports for them, such as ray-client-server-port for the head node.

But for the ray worker nodes, I am not sure which ports should be set. As the document shows, the dashboard-agent-listen-port should be the random value by default.

But when I deployed my ray job , I found that two ray worker nodes on a kubernetes node were using the same port 52365 for dashboard-agent-listen-port :

And this prevented me from viewing a node's logs from the dashboard, when I set different dashboard-agent-listen-port for ray worker nodes, this problem can be solved.

So should I set different ports for each in the ray start params for ray worker nodes ? My guess is that dashboard-agent-grpc-port and other ports like node-manager-port will have similar issues.

Reproduction script

apiVersion: ray.io/v1alpha1
kind: RayJob
metadata:
  # too long name may cause ray cluster reconcile error
  name: job-without-autoscaler
  namespace: default
spec:
  entrypoint: python /home/ray/samples/sample_code.py  # shutdownAfterJobFinishes specifies whether the RayCluster should be deleted after the RayJob finishes. Default is false.
  shutdownAfterJobFinishes: true

  # ttlSecondsAfterFinished specifies the number of seconds after which the RayCluster will be deleted after the RayJob finishes.
  ttlSecondsAfterFinished: 3600

  # RuntimeEnvYAML represents the runtime environment configuration provided as a multi-line YAML string.
  # See https://docs.ray.io/en/latest/ray-core/handling-dependencies.html for details.
  # (New in KubeRay version 1.0.)

  # base64-encoded string of the runtime env json string.
  runtimeEnv: "eyJjb3VudGVyX25hbWUiOiAidGVzdF9jb3VudGVyIn0="
  # Suspend specifies whether the RayJob controller should create a RayCluster instance.
  # If a job is applied with the suspend field set to true, the RayCluster will not be created and we will wait for the transition to false.
  # If the RayCluster is already created, it will be deleted. In the case of transition to false, a new RayCluster will be created.
  # suspend: false

  # rayClusterSpec specifies the RayCluster instance to be created by the RayJob controller.
  rayClusterSpec:
    rayVersion: '2.7.1' # should match the Ray version in the image of the containers
    # Ray head pod template
    headGroupSpec:
      # The `rayStartParams` are used to configure the `ray start` command.
      # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
      # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
      rayStartParams:
        dashboard-host: "0.0.0.0"
        node-ip-address: $MY_POD_IP
        ray-client-server-port: "61001"
        dashboard-port: "61265"
        port: "61379"
        metrics-export-port: "61080"  # not random by default on k8s, default to 8080
        include-dashboard: "true"
        min-worker-port: "10002"   # set random port range
        max-worker-port: "19999"
        # 52365 for default
        dashboard-agent-listen-port: "62365"
        num-cpus: "0"  # prevent scheduling workloads to head
      #pod template
      template:
        spec:
          hostNetwork: true # host network
          dnsPolicy: ClusterFirstWithHostNet
          nodeSelector: {}
          containers:
            - name: ray-head
              # image: docker.io/rayproject/ray-ml:2.7.1.9f07c1-py38
              image:  docker.io/rayproject/ray:4e2b52-aarch64
              env:
                - name: MY_POD_IP
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
              # Keep this preStop hook in each Ray container config.
              lifecycle:
                preStop:
                  exec:
                    command: [ "/bin/sh","-c","ray stop" ]
              ports:
                - containerPort: 61379
                  name: gcs # gcs
                - containerPort: 61265
                  name: dashboard
                - containerPort: 61001
                  name: head
                - containerPort: 61080  # not random by default on k8s
                  name: metrics
                - containerPort: 62365
                  name: listen
              resources:
                limits:
                  cpu: 2
                  memory: 2Gi
                requests:
                  cpu: 2
                  memory: 2Gi
              volumeMounts:
                - mountPath: /home/ray/samples
                  name: code-sample
          volumes:
            # You set volumes at the Pod level, then mount them into containers inside that Pod
            - name: code-sample
              configMap:
                # Provide the name of the ConfigMap you want to mount.
                name: ray-job-code-sample
                # An array of keys from the ConfigMap to create as files
                items:
                  - key: sample_code.py
                    path: sample_code.py
    workerGroupSpecs:
      # the pod replicas in this group typed worker
      - replicas: 2
        minReplicas: 2
        maxReplicas: 5
        # logical group name, for this called small-group, also can be functional
        groupName: small-group
        # The `rayStartParams` are used to configure the `ray start` command.
        # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
        # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
        rayStartParams:
          node-ip-address: $MY_POD_IP
          metrics-export-port: "61081"  # not random by default on k8s, default to 8080
          min-worker-port: "10002"
          max-worker-port: "19999"
          dashboard-agent-listen-port: "62366"
        #pod template
        template:
          spec:
            hostNetwork: true # host network
            dnsPolicy: ClusterFirstWithHostNet
            nodeSelector: {}
            containers:
              - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
                # image: docker.io/rayproject/ray-ml:2.7.1.9f07c1-py38
                image:  docker.io/rayproject/ray:4e2b52-aarch64
                env:
                  - name: MY_POD_IP
                    valueFrom:
                      fieldRef:
                        fieldPath: status.podIP
                lifecycle:
                  preStop:
                    exec:
                      command: [ "/bin/sh","-c","ray stop" ]
                ports: # Optional service port overrides
                  - containerPort: 8080
                    name: metrics
                  - containerPort: 62366
                    name: listen
                resources:
                  limits:
                    cpu: 1
                    memory: 1Gi
                  requests:
                    cpu: 1
                    memory: 1Gi
                volumeMounts:
                  - mountPath: /home/ray/samples
                    name: code-sample
            volumes:
              # You set volumes at the Pod level, then mount them into containers inside that Pod
              - name: code-sample
                configMap:
                  # Provide the name of the ConfigMap you want to mount.
                  name: ray-job-code-sample
                  # An array of keys from the ConfigMap to create as files
                  items:
                    - key: sample_code.py
                      path: sample_code.py
      - replicas: 2
        minReplicas: 2
        maxReplicas: 5
        # logical group name, for this called small-group, also can be functional
        groupName: small-group2
        # The `rayStartParams` are used to configure the `ray start` command.
        # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay.
        # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`.
        rayStartParams:
          node-ip-address: $MY_POD_IP
          metrics-export-port: "61081"  # not random by default on k8s, default to 8080
          min-worker-port: "10002"
          max-worker-port: "19999"
          dashboard-agent-listen-port: "62367"
        #pod template
        template:
          spec:
            hostNetwork: true # host network
            dnsPolicy: ClusterFirstWithHostNet
            nodeSelector: { }
            containers:
              - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
                # image: docker.io/rayproject/ray-ml:2.7.1.9f07c1-py38
                image: docker.io/rayproject/ray:4e2b52-aarch64
                env:
                  - name: MY_POD_IP
                    valueFrom:
                      fieldRef:
                        fieldPath: status.podIP
                lifecycle:
                  preStop:
                    exec:
                      command: [ "/bin/sh","-c","ray stop" ]
                ports: # Optional service port overrides
                  - containerPort: 8081
                    name: metrics
                  - containerPort: 62367
                    name: listen
                resources:
                  limits:
                    cpu: 1
                    memory: 1Gi
                  requests:
                    cpu: 1
                    memory: 1Gi
                volumeMounts:
                  - mountPath: /home/ray/samples
                    name: code-sample
            volumes:
              # You set volumes at the Pod level, then mount them into containers inside that Pod
              - name: code-sample
                configMap:
                  # Provide the name of the ConfigMap you want to mount.
                  name: ray-job-code-sample
                  # An array of keys from the ConfigMap to create as files
                  items:
                    - key: sample_code.py
                      path: sample_code.py
  # SubmitterPodTemplate is the template for the pod that will run the `ray job submit` command against the RayCluster.
  # If SubmitterPodTemplate is specified, the first container is assumed to be the submitter container.
  submitterPodTemplate:
     spec:
       nodeSelector: {}
       restartPolicy: Never
       containers:
         # - name: custom-rayjob-submitter-pod
         - name: ray-submitter
           # image:  docker.io/rayproject/ray-ml:2.7.1.9f07c1-py38
           image:  docker.io/rayproject/ray:4e2b52-aarch64
  #         # If Command is not specified, the correct command will be supplied at runtime using the RayJob spec `entrypoint` field.
  #         # Specifying Command is not recommended.
  #         # command: ["ray job submit --address=http://rayjob-sample-raycluster-v6qcq-head-svc.default.svc.cluster.local:8265 -- echo hello world"]
######################Ray code sample#################################
# this sample is from https://docs.ray.io/en/latest/cluster/job-submission.html#quick-start-example
# it is mounted into the container and executed to show the Ray job at work

Anything else

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

kevin85421 commented 10 months ago

So should I set different ports for each in the ray start params for ray worker nodes ?

I think so.

Out of curiosity, why do you decide to use the host network mode? If you must use this mode, I suggest running only one Ray Pod on a single Kubernetes node. There could be many issues if multiple Pods on a single Kubernetes node share the same network namespace.

surenyufuz commented 10 months ago

So should I set different ports for each in the ray start params for ray worker nodes ?

I think so.

Out of curiosity, why do you decide to use the host network mode? If you must use this mode, I suggest running only one Ray Pod on a single Kubernetes node. There could be many issues if multiple Pods on a single Kubernetes node share the same network namespace.

Thanks for your reply.

Due to the container port scarcity in my Kubernetes network environment, I have to use the host network mode to deploy ray jobs as many as possible.

Running only one Ray Pod on a single Kubernetes node may be not suitable for a common Kubernetes clusters, because other workloads may exist, but thank you anyway.

By the way, the document described the dashboard-agent-listen-port settings may be not correct, as the Ray start script shows:

dashboard-agent-listen-port would be set a default value, rather than a random one，it bothers me a lot. If the document could be fixed, this issue will have nothing to do with bugs, thanks.

kevin85421 commented 9 months ago

Would you mind opening a PR in the Ray repository to update the doc? You can also cc me there.

surenyufuz commented 9 months ago

Would you mind opening a PR in the Ray repository to update the doc? You can also cc me there.

My pleasure, I have raised a related PR https://github.com/ray-project/ray/pull/42060 .

ray-project / kuberay