The GetRayJobStatus() function part of the go routine MonitorRunningJobs fetches all the jobs with running status from the database and uses the ray job ID (submission ID) to fetch the status of the job. However, after we deploy a new version of the base ray job on testeks, it loses the previous state, and returns a 404 job not found to the gateway. This was causing a CrashLoopBackoff on the backend container. To fix this, now the GetRayJobStatus() handles this scenario properly and marks the job as failed and moves on instead of crashing.
What type of PR is this?
Description
The GetRayJobStatus() function part of the go routine MonitorRunningJobs fetches all the jobs with running status from the database and uses the ray job ID (submission ID) to fetch the status of the job. However, after we deploy a new version of the base ray job on testeks, it loses the previous state, and returns a 404 job not found to the gateway. This was causing a CrashLoopBackoff on the backend container. To fix this, now the GetRayJobStatus() handles this scenario properly and marks the job as failed and moves on instead of crashing.
After deploying to test:![image](https://github.com/user-attachments/assets/950af284-c70c-498b-9602-69e411cc72f8)