add distinct logging for sources

[goodenou]

Address issue 650:

Separate log files are now created for each source. Source log info is still added to the decisionengine_structlog_debug.log file, along with all other logging. The name of the logger is "source" (compare with "decisionengine" and "channel"). The "module" items in the log entries are the names of the sources, as given in the config file.

I ran tests using the channels job_classification and resource_request. The following files were created and populated: decisionengine_debug.log decisionengine.log decisionengine_structlog_debug.log FactoryEntriesSource.log FactoryGlobalManifests.log FigureOfMerit.log GceFigureOfMerit.log JobCategorizationSourceProxy.log job_classification.log JobManifestsSourceProxy.log jobs_manifests.log NerscFigureOfMerit.log resource_request.log StartdManifestsSource.log

An example log entry from decisionengine_structlog.log is: {"channel": "job_classification", "event": "Starting source loop for jobs_manifests", "level": "info", "logger": "source", "module": "jobs_manifests", "timestamp": "2022-05-27 18:51:32"}

An example log entry from FactoryEntriesSource.log is: 2022-05-27 18:51:33,259 - source - SourceWorkers - INFO - {"channel": "resource_request", "event": "Starting source loop for FactoryEntriesSource", "level": "info", "logger": "source", "module": "FactoryEntriesSource", "timestamp": "2022-05-27 18:51:33"}

[codecov[bot]]

Codecov Report

Merging #654 (c627b1f) into master (2b94604) will increase coverage by 0.28%. The diff coverage is 93.02%.

@@            Coverage Diff             @@
##           master     #654      +/-   ##
==========================================
+ Coverage   96.94%   97.23%   +0.28%     
==========================================
  Files          49       49              
  Lines        2947     3036      +89     
  Branches      402      486      +84     
==========================================
+ Hits         2857     2952      +95     
+ Misses         60       56       -4     
+ Partials       30       28       -2     

Flag	Coverage Δ
python-3.10	97.06% <93.02%> (?)
python-3.6	96.82% <93.02%> (-0.13%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../decisionengine/framework/engine/DecisionEngine.py	92.67% <80.95%> (-0.41%)	:arrow_down:
...c/decisionengine/framework/engine/SourceWorkers.py	97.77% <96.29%> (-0.75%)	:arrow_down:
src/decisionengine/framework/engine/de_client.py	100.00% <100.00%> (ø)
...sionengine/framework/modules/logging_configDict.py	100.00% <100.00%> (ø)
src/decisionengine/framework/modules/Transform.py	100.00% <0.00%> (ø)
...ionengine/framework/taskmanager/ProcessingState.py	100.00% <0.00%> (ø)
...rc/decisionengine/framework/dataspace/dataspace.py	93.97% <0.00%> (+0.07%)	:arrow_up:
...aspace/datasources/sqlalchemy_ds/datasource_api.py	100.00% <0.00%> (+1.22%)	:arrow_up:
src/decisionengine/framework/dataspace/maintain.py	100.00% <0.00%> (+2.97%)	:arrow_up:
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2b94604...c627b1f. Read the comment docs.

[pep8speaks]

Hello @goodenou! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2022-06-16 20:47:46 UTC

[vitodb]

I ran unit test for this PR on Jenkins yesterday, it failed on FAILED src/decisionengine/framework/dataspace/tests/test_Reaper.py::test_source_fail_can_be_fixed

After an update of the PR this morning, I ran again unit tests on Jenkins, this time all was fine

I did run unit test on my VM merging this PR to the current DE master, which includes also PR#657 not merged into this PR, but I think changed in 657 are not affecting the test. Running the unit tests using the Docker image and, once I made sure to have all required dependencies and python modules installed, running unit test directly on the VM, there were no failures reported.