Make `logging.yml` read by default

[noklam]

Quick summary:

• :white_check_mark:There are at least one obvious user problem - which is logging get chosen as a tool but has no effect, and the consensus is that we should fix it. - https://github.com/kedro-org/kedro/issues/3446
  • One possible solution is hard coding the path to conf/logging.yml with option to override with KEDRO_LOGGINGI_CONFIG, very similar to https://github.com/kedro-org/kedro/commit/be3d28d981ecf51d8fef96a256d9c0ef74d5d8c4
• It's unclear whether we should do more for logging, such as adding a new -v flag
• It's mentioned logging in development and logging in production are very different, and users probably have different needs:
  • It was mentioned we may want to de-couple rich as a plugin, pip-uninstallable to deactivate it or at least provide an easy enough way to opt out from it.
• We may consider improving the messaging of logging and people should seek for production grade logging.
• It was mentioned even though there are users need, Kedro can still say NO as we had for many other things.

Antony shared the old user research here in case you miss it at the end: https://github.com/kedro-org/kedro/issues/2281#issuecomment-1471805422

Originally posted by @noklam in https://github.com/kedro-org/kedro/discussions/3755#discussioncomment-9009731

Context

Since 0.19, we introduced a few changes:

• 2637

• 2535

• A new kedro project creation flow, i.e. kedro new --tools (with logging)

This create a bad UX because:

• New users expect logging.yml will be used by default
• Existing users migrate from 0.18 -> 0.19 things start working unexpectedly

The scope of ticket is to implement a fix to make sure the default file is read. KEDRO_LOGGING_CONFIG will take priority when it is provided. (See https://github.com/kedro-org/kedro/commit/be3d28d981ecf51d8fef96a256d9c0ef74d5d8c4 for inspiration)

Solution

https://github.com/kedro-org/kedro/commit/be3d28d981ecf51d8fef96a256d9c0ef74d5d8c4 is a good starting point. At the minimal case, the happy path of

• kedro new --tools=log
• kedro run should auotmatically read logging.yml

There are things to consider:

• Should we consider accepting logging* or searching for subfolder?
• We need to consider corner case where logging.yml is at CONF_SOURCE/logging.yml, by default CONF_SOURCE is conf but user has the option to change it. (Not handled in the old PR)
  • You may find that you cannot access CONF_SOURCE because it is not read yet when logging is initialised, does it takes a lot of changes to make it possible? Is there workaround?

#

[SajidAlamQB]

Completed in https://github.com/kedro-org/kedro/pull/3831