search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.91k stars 900 forks source link

Consider `kedro run -v` verbose mode #3591

Closed noklam closed 6 months ago

noklam commented 8 months ago

Description

The current logging configuration is too granular. We should use root logger only. The exception would be info_logger or error_logger because they are mean to capture that specific log level.

The most common case is that I want to see all the logs during debugging. In production case (packaged project), how do I even change it to see the log if I want to? Editing the configuration file doesn't seem to be a good idea.

Inspired by pytest -v pytest -vv Proposal:

  1. kedro run -v verbose mode - The Project logging level will be set to DEBUG because this should be the most common case, I want to see the log message that I wrote in the project
  2. kedro run -vv more verbose mode - Everything should be shown now, including any plugin logging or kedro logs

Context

It's hard to get logging right, in most case I either expect seeing just the default level or I want to see everything during debugging. a -v flag comes in handy and is a common convention among many CLI tools. i.e. pytest.

Possible Implementation

Add new argument in the CLI, and override the level of logging in LOGGING.

Possible Alternatives

Edited 2024-03-07 I have splited this ticket to two tickets

noklam commented 7 months ago

Putting a note here to remind myself to do a demo when we groom this.

astrojuanlu commented 7 months ago

I've said it in https://github.com/kedro-org/kedro/issues/3446#issuecomment-1936278169, but for the record, +1 to this. From https://12factor.net/logs:

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.

astrojuanlu commented 7 months ago

Another point in favour of having kedro -vv: sometimes you just want to know what kedro is doing without having to configure the whole stdlib logging. This is how most other tools work.

Here is an example of a user that's puzzled because kedro just hangs https://linen-slack.kedro.org/t/16630828/i-m-encountering-issues-when-running-the-load-ext-kedro-ipyt#3758284a-34fb-435c-b054-dd3713cd78c9