[Serve] Make it easier to view application-logs (aka logs that I write) in the ray-dashboard

[JakeSummers]

Description

When I am developing, I would like to debug my application by easily viewing my application's logs in the ray-dashboard. Right now the application logs are intermixed with the core ray logs.

Full discussion in this slack thread: https://ray-distributed.slack.com/archives/CNCKBBRJL/p1699328167338449

Use case

Currently I log things in ray-serve like this:

# At the top of your file: 
import logging
logger = logging.getLogger("ray.serve")

# When you want to log something: 
logger.info("Hi from call - logger")

As per the instructions here:

• Ray Serve / Monitor Your Application / Ray Logging
• and Monitoring and Debugging / User Guides / Configuring Logging

This ends up being shown in the ray-serve dashboard like this:

There are several issues with this:

1. Key Issue: it is hard for me to find my application level logs distinctly from the ray system level logs
2. Secondary Issue: There is a lot of redundant info in the logs. In the ray dashboard, I on a page for a specific deployment, so I don't need the deployment's name and instance to be printed on every line

Possible Solution

One solution, to allow me to filter for just my application level logs would be to print the full path of the file:

Current: INFO 2023-11-06 18:55:14,971 app_Pipeline app_Pipeline#StyBUT KkTVnifWpg /http-endpoint app_stuff file.py:105 - Hi from call - logger 
Possible: INFO 2023-11-06 18:55:14,971 app_Pipeline app_Pipeline#StyBUT KkTVnifWpg /http-endpoint app_stuff my_package.file:105 - Hi from call - logger 

This would allow my to filter the UI for my_package:

[scottsun94]

On the dashboard side, we can potentially filter out some of the repeated metadata when fetching the logs. cc: @alanwguo

[scottsun94]

Before we are able to fix this issue on dashboard side, a workaround will be customizing the formatter of the serve logger and remove the redundant or add metadata you want to? @JakeSummers

[JakeSummers]

RE @scottsun94

Before we are able to fix this issue on dashboard side, a workaround will be customizing the formatter of the serve logger and remove the redundant or add metadata you want to? @JakeSummers

I looked briefly into how to do that, but it wasn't obvious to me. Can you point me in the right direction?

[scottsun94]

RE @scottsun94

Before we are able to fix this issue on dashboard side, a workaround will be customizing the formatter of the serve logger and remove the redundant or add metadata you want to? @JakeSummers

I looked briefly into how to do that, but it wasn't obvious to me. Can you point me in the right direction?

@sihanwang41 do you have any pointer? Or what's the best way to do it?

[sihanwang41]

hi @JakeSummers , For filtering out ray serve internal logs by requests, ray serve introduce the new logging api (will be out to 2.9). After enable the flag, you are able to disable all the serve internal logs by requests, so that it will be eaiser to check your application log.