Implement logging feature

[richardogoma]

Here are some considerations:

1. Create a logs/ directory
2. Create a log submodule, most likely in the load submodule of the etl module. This module has the following features: i. Contains functions to write the INFO and ERROR messages to a program.log file in the logs/ dir ii. Contains a default script that runs when the module is invoked. This script checks if the KiB of the log file exceeds 1024KiB (or a certain threshold), and if so, it rotates the log file. The old log file would be added to an archive (rar or zip format) in the logs/ dir

The log submodule would have two functions: i. log.Info(info_msg) which would handle the info messages ii. log.Error(error_msg) which would handle the error messages

from etl.load import log

log.Info(
    f"{datetime.now()}: INFO: Inserted {info} into the database. ETL process duration: {process_duration} secs"
)

log.Error(f"{datetime.now()}: ERROR: {str(error_msg)}")

3. In the __init__.py file of the etl module, the logging functions should be invoked appropriately in place of the print function currently in use

[richardogoma]

See this article for insights

Normally you might write to a log file like this:

log.Debug("Incoming metrics data"); This would produce a line like this in your log:

DEBUG 2017-01-27 16:17:58 – Incoming metrics data

This gives insight into the logging function design and the log output.

[richardogoma]

See this article by Khuyen Khan. In fact implement this feature around it: Loguru: Simple as Print, Flexible as Logging

Logging in Python: A Developer’s Guide