Kraysent / OMTool

Modeling N-Body problem in galactic evolution application.
https://omtool.readthedocs.io/en/latest/
Apache License 2.0
0 stars 0 forks source link

Write all logs in JSON format #78

Closed Kraysent closed 2 years ago

Kraysent commented 2 years ago

Problem: logging is not flexible enough. If user wants to log some tasks only into file and not in console and some other tasks only into console and not into file, they cannot do this. Also ther is json logging which requires configuration-time description of fields.

Possible solution: split onle logger object instance into multiple number of separate ones. Each one has only one handler and ID. When one wants to log some task as JSON they specify ID of logger and possibly some params.

Kraysent commented 2 years ago

Probably the most versatile solution would be to extract this logic into whole separate repository and pip package. This package would provide functions to create specific handlers (console, file, json) and functions for logging (info, error, warning, debug).

Kraysent commented 2 years ago

Repository is located here.

For now it has methods to create loggers of specific type and logging methods.

Kraysent commented 2 years ago

In order to use it, one needs to read the configuration from YAML and put it into some configuration class. Next step would be to create some initialize function where  configuration dictionary would be converted into add_<some>_logger function calls.

Apart from that, one needs to list some "standard" loggers. They should be used in all cases if not stated otherwise.

Kraysent commented 2 years ago

Or maybe it would be just easier to get rid of non-JSON logging entirely and make all logs in this format. Downside of this approach would be slightly less readable log messages but I think this is manageable since it would make logging code significantly simpler.

Kraysent commented 2 years ago

First step would be to decouple logger from the actual program so it might be swapped easily.

Kraysent commented 2 years ago

Next step is to introduce special field data which would store arbitrary values that are provided into info functions. To identify the structure of this data, one more field should be introduced: message_type. By default all the logs have "message" type.

So when called info('Some test informational message') output JSON would have structure:

{
  ....
  "message_type": "msg",
  "data": {
    "message": "Some test informational message"
  }
}

Otherwise, when called info(message_type = 'task_data', data = {"x": 1, "y": 2}) output JSON would have following structure:

{
  ....
  "message_type": "task_data",
  "data": {
    "x": 1,
    "y": 2
  }
}
Kraysent commented 2 years ago

One of final touches would be to write date and time in custom format.