[Question] Logging ray errors

[umutcakan]

Hello,

I observed there is a CLI logger that outputs information regarding trial processes, even if worker dies before it even starts it gives some information regarding to that. In my case, I want to use a logger to see logs in a separate place, so I tried to create one class that inherits from ray.tune.logger.loggerCallback, and sent it as an argument to 'loggers' parameters and can use different methods to log trial processes and it was successful so far.

My question is, is there a way to log ray tune errors with this loggerCallback or should I use another approach? (e.g. I want to log if there is an error with worker initialization). I prefer to use a custom logger since the process will mostly work remotely, and accessing files one by one is not feasible in the future.

Thanks in advance.

[Yard1]

Do you have an example of the messages you would want to log?

[umutcakan]

Hey again,

Unfortunately, I could not replicate the error I was getting before, previously one of the errors was "Worker died unexpectedly while executing this task." Right now I can manage to make it work on local runs, but there is a problem when it is deployed and basically I want to see the problem in the remote server to debug the problem or understand if it is related to ray workers

[Yard1]

Those sort of logs should be written by Ray Core and saved in the session logs on the node. See https://docs.ray.io/en/latest/ray-observability/ray-logging.html

[umutcakan]

Thank you, I will look into that!