Unified logging configuration/strategy for KAT's modules

[Darwinkel]

Did some experimenting in https://github.com/minvws/nl-kat-coordination/pull/1314 as part of https://github.com/minvws/nl-kat-coordination/issues/1299. It would be nice if there was a single way to configure logging, maybe through .env or by refactoring logging.json.

Up for discussion

We should answer the following questions:

• Which logging format do we want to use? Either logging.conf, logging.json or logging.yml.
• Do we add a separate file with a production setup for each service that we use when DEBUG=False?

TODOs

• [ ] Rocky should have configurable logger support
• [ ] Octopoes should use json, or the other modules should use yml
• [ ] There should be instructions or a guide on how to modify KAT's logging configuration
• [ ] Fix spacing, conventions, and formatting among all files
• [ ] Bytes has logging.conf and is the odd one out
• [ ] There should be two logging configs out of the box for each module: we default to production logging, but switch to the debug config when DEBUG is set
  • [ ] Unless a user has specifically set a log file and deviates from the default, then we don't do this
  • [ ] Defaulting to production-grade logging should also partially resolve the problem that some users have that KAT eats too much disk space

[Darwinkel]

Additionally, this seems like the right moment to write some policy about what each logging level means in KAT (and which one should be enabled on production setups).

Some thoughts by @Donnype and me:

Included in debugging mode:

• DEBUG: Detailed information, typically only of interest to a developer trying to diagnose a problem.
• INFO: Confirmation that things are working as expected under normal circumstances.

Included in production mode:

• WARNING: An indication that something unexpected happened, or that a problem might occur in the near future (e.g. ‘disk space low’). The software is still working as expected.
• ERROR: Due to a more serious problem, the software has not been able to perform some function.
• CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

In some applications, DEBUG is extremely verbose, and may include all connections and queries in e.g. sqlalchemy. This may make the logs completely unreadable. That's probably not desirable either. We'll have to think about how we propagate/configure logging settings to our libraries.

[Donnype]

Level	Description	Examples	Context
DEBUG	Detailed info, typically only of interest to developers diagnosing a problem.	Database queries fine-grained control-flow Persisting objects	Development
INFO	Confirmation the software is operating as expected.	API calls Request life cycles Task life cycles	Development Production
WARNING	Info that needs attention: a problem might be happening or occur in the future. The software is still operational.	Inconsistencies Invalid input HTTP 400 responses Deprecation of services Retries of operations Disk space low	Development Production
ERROR	Due to a serious, unexpected problem, the software has not been able to perform an operation.	Exceptions Failing to persist data Unexpected behaviour	Development Production
CRITICAL	A serious error, indicating that the program itself may be unable to continue running.	Crashes (Database) Services being down	Development Production Alerting

[dekkers]

I was actually also thinking about the levels when I saw lots of INFO messages yesterday but also didn't think they should be DEBUG level. I think the isssue is that if we want to log what is currently going on that doesn't indicate a problem we only have INFO. But that makes INFO very verbose. I think what we are missing is a level in between INFO and DEBUG, something we could call VERBOSE.

For example logging every request should not be debug level I think, but logging them with INFO means we log a lot of lines on big installation. If we would log it on a VERBOSE level then people can decide whether they want that or not.

It is possible to add custom levels with python logging: https://docs.python.org/3/library/logging.html#logging.addLevelName