In ray 2.3, there is PR to get real job id from log file content instead of log file name. Check log_monitor.py. If there is not job id (pattern: ":job_id:") found, then job_id field of log message object is None. For log message without job_id set, ray just blindly prints it out in console.
After adding below code snippet in "premain" method of Java agent introduced in #318 , raydp just performs consistently across different ray versions. No executor logs being flushed to console.
jiafuzha
commented
1 year ago
In ray 2.3, there is PR to get real job id from log file content instead of log file name. Check log_monitor.py. If there is not job id (pattern: ":job_id:") found, then job_id field of log message object is None. For log message without job_id set, ray just blindly prints it out in console.
After adding below code snippet in "premain" method of Java agent introduced in #318 , raydp just performs consistently across different ray versions. No executor logs being flushed to console.
String jobId = System.getenv("RAY_JOB_ID"); String rayAddress = System.getProperty("ray.address"); if (jobId != null && rayAddress != null) { try (FileWriter writer = new FileWriter(logDir + "/java-worker-" + jobId + "-" + pid + ".log")) { writer.write(":job_id:" + jobId + "\n"); } }