addition of queue logging for multiprocessing

[goodenou]

All loggers, "decisionengine" and "channel" loggers now operate using a QueueHandler that stashes the messages into the logging queue.

There is one QueueHandler, with one corresponding multiprocessing queue that is sends log messages to. All loggers have this handler attached. (They also have their standard file handlers attached.) There is one QueueListener that monitors the logging queue for log messages, and sends them to the "DEBUG" level file handler with structlog formatting.

IMPORTANT NOTE: The QueueLogger causes all kinds of trouble if enabled during unit tests. Therefore, I have added an additional configuration parameter to the "logging' section of the config dictionary, "start_q_logger". This parameter should be set to "True" for any configuration file that is running a job and "False" for any configuration file that is doing a unit test.

[pep8speaks]

Hello @goodenou! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/decisionengine/framework/modules/tests/test_QueueLogger.py:

Line 4:1: F401 'gc' imported but unused

Comment last updated at 2021-12-13 18:25:38 UTC

[codecov[bot]]

Codecov Report

Merging #563 (d3fc91a) into master (77dd8d5) will increase coverage by 0.23%. The diff coverage is 92.40%.

@@            Coverage Diff             @@
##           master     #563      +/-   ##
==========================================
+ Coverage   83.97%   84.20%   +0.23%     
==========================================
  Files          45       46       +1     
  Lines        2702     2748      +46     
  Branches      484      492       +8     
==========================================
+ Hits         2269     2314      +45     
+ Misses        391      390       -1     
- Partials       42       44       +2     

Flag	Coverage Δ
python-3.10	84.13% <92.40%> (+0.41%)	:arrow_up:
python-3.6	83.69% <92.40%> (+0.05%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/decisionengine/framework/modules/de_logger.py	91.48% <87.09%> (-8.52%)	:arrow_down:
src/decisionengine/framework/engine/Workers.py	76.62% <88.23%> (+6.62%)	:arrow_up:
...ionengine/framework/config/ChannelConfigHandler.py	94.66% <100.00%> (ø)
.../decisionengine/framework/engine/DecisionEngine.py	36.30% <100.00%> (+0.27%)	:arrow_up:
...rc/decisionengine/framework/modules/QueueLogger.py	100.00% <100.00%> (ø)
...sionengine/framework/modules/logging_configDict.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 77dd8d5...d3fc91a. Read the comment docs.

[lgtm-com[bot]]

This pull request introduces 3 alerts when merging e5716868ca25a4bc95684cbd6fe5868cfb639132 into 82faa27127aee458e40bc7e491f411a9b9c10727 - view on LGTM.com

new alerts:

• 2 for Unused local variable
• 1 for Unused import