[Dashboard][Observability] Dashboard shows that a task is still running after a RayCluster with GCS FT restarts

[kevin85421]

What happened + What you expected to happen

1. Create a RayCluster with GCS fault-tolerance.
2. Execute a Python script on the head Pod to submit a task to a worker Pod.
3. Kill the GCS server process on the head Pod
4. Head Pod will be terminated by itself after gcs_rpc_server_reconnect_timeout_s seconds (60s by default).
5. Worker Pods are still running, and the task is still running for a while until the driver process on the old head Pod is killed.
6. New head Pod becomes ready. Connect to the dashboard, and the dashboard shows that the task is still running.

• In my understanding, it makes sense that the task stops when the driver process dies because the driver process is the owner of the task. The Ray architecture paper does not explicitly mention this, so I am not 100% sure.

• I monitored the task's stdout log (e.g. /tmp/ray/session_latest/log/worker-aaaaa-ffffffff-123.out) on the worker Pod to confirm that the task had stopped.

There is no log / task / actor for the job, but the status is "RUNNING".

Versions / Dependencies

• Ray 2.3.0
• KubeRay v0.5.0

Reproduction script

• raycluster-ft.yaml
• test.py

# Create a Kubernetes cluster
kind create cluster --image=kindest/node:v1.23.0

# Install a KubeRay operator
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0

# Create a RayCluster with GCS FT
kubectl apply -f raycluster-ft.yaml

# Run test.py in the head Pod
python3 test.py

# Kill the GCS server process in the head Pod
pkill gcs_server

# (1) Wait for the new head to be ready.
# (2) Monitor the task's STDOUT log
# (3) Check the dashboard when the new head is ready.

Issue Severity

None

[kevin85421]

In my understanding, it makes sense that the task stops when the driver process dies because the driver process is the owner of the task. The Ray architecture paper does not explicitly mention this, so I am not 100% sure.

@iycheng is this correct? Thanks!

[scottsun94]

cc: @rkooo567 @rickyyx to investigate and see whether this is an easy fix for @iycheng

[llidev]

Hi @kevin85421 @iycheng

I wonder if there is any update on this issue? IMO this is more than an observability-ux problem. Job level FT is not well documented, and it's very confusing for user about how to develop resilient Ray job applications...

[rickyyx]

I will take a look in the coming sprint! If that's not too late.

[llidev]

Thanks @rickyyx I just created two new issues related to GCS FT: https://github.com/ray-project/ray/issues/38786 https://github.com/ray-project/ray/issues/38785 hope they are related!

[rkooo567]

The task information is not working with GCS HA now (basically all task data is stored in memory). It is actually very tricky to fix it because task information adds a lot of pressure to the storage (Redis) if we start persisting them in Redis storage.