Improve logging configuration experience

[morhidi]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

There's been numerous useful improvements added lately to both Ray & Kuberay that enhances the overall logging experience for users, end we would like to continue on this route with this proposal. We have the following high level concepts in mind:

• KubeRay operator should provide reasonable default settings for Ray process logging exposed in generic config file format. For example :

  formatters:
  brief:
  # configuration for formatter with id 'brief' goes here
  precise:
  # configuration for formatter with id 'precise' goes here
  handlers:
  h1: #This is an id
  # configuration of handler with id 'h1' goes here
  formatter: brief
  h2: #This is another id
  # configuration of handler with id 'h2' goes here
  formatter: precise
  loggers:
  foo.bar.baz:
  # other configuration for logger 'foo.bar.baz'
  handlers: [h1, h2]

• The default should contain two loggers:
  • File logger which is also exposed on the dashboard in a human readable format
  • Console logger that exposed on Kubernetes container in human readable format
• Logging customization should be available as an optional field at CR level for users to easily override the defaults
  • e.g. changing logging level on existing loggers (file/console)
  • e.g. define new file based structured loggers for centralized logging solutions
• For the sake of simplicity users should provide the defaults + customization together when overriding defaults cc @kevin85421

Use case

These ideas are inspired by Apache Flink Kubernetes operator, we're trying to propose a similar concept https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

[hahahannes]

Hi! Would this also include the possibility to configure the log level of the kuberay operator? So far I could not find a configuration option for this.