Open spolcyn opened 2 years ago
I think @clarkzinzow @rkooo567 are the RAY_LOG_TO_STDERR
experts.
I think @edoakes is on-call for jobs stuff.
Hmm this is pretty tricky. It is because how we display logs to the driver is by reading a log file. If RAY_LOG_TO_STDERR
is set, we don't write anything to the log file, so it won't be able to get logs from the driver. I think there are 2 ways to fix it, but both are not very straightforward nor clean.
RAY_LOG_TO_STDERR
is set, the component directly publishes the batched log message to the driver. This can cause some performance regression when RAY_LOG_TO_STDERR
is set because it will cause the higher load to the driver + workers. Hi, I'm a bot from the Ray team :)
To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.
If there is no further activity in the 14 days, the issue will be closed!
You can always ask for help on our discussion forum or Ray's public slack channel.
Keep open.
What happened + What you expected to happen
RAY_LOG_TO_STDERR
is set, a job doesn't receive the logs from actors it creates.RAY_LOG_TO_STDERR
is set or unset.Note: "empty" here means the job driver log lacks all the previous output from the actors created, and it only has the internal
python-core-driver
cpp logs.Potentially related to https://github.com/ray-project/ray/issues/23564
Versions / Dependencies
Ray 1.12.0 Python 3.7.12 Ubuntu 20.04
Reproduction script
Instructions:
test_job.py
andrun.py
in the same folderpython test_job.py
repro-logs.txt
, findsearch for me in logs
string presentRAY_LOG_TO_STDERR=1 python test_job.py
repro-logs.txt
do not findsearch for me in logs
string presentIssue Severity
Medium: It is a significant difficulty but I can work around it.