pytroll / trollflow2

Next generation Trollflow. Trollflow is for batch-processing satellite data using Satpy
https://trollflow2.readthedocs.org/
GNU General Public License v3.0
10 stars 15 forks source link

Logging fix using global QueueHandler #124

Closed mraspaud closed 3 years ago

mraspaud commented 3 years ago

It has been observed that trollflow2 v0.11.0 hangs quite often, probably due to a race condition in the logging code.

It is very likely that this bug has been introduced in #115, hence this attempt to make the QueueHandler usage more streamlined, independently of whether trollflow2 runs on threads or subprocesses, and more importantly by setting up the queued log handler and other handlers prior to running the main loop.

EDIT: for completeness, it should be noted that many live tests have been run with v0.10.0 at SMHI and that hanging specifically in the logging part was also happening.

codecov[bot] commented 3 years ago

Codecov Report

Merging #124 (1a63f48) into main (582852b) will decrease coverage by 0.39%. The diff coverage is 91.78%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #124      +/-   ##
==========================================
- Coverage   95.10%   94.70%   -0.40%     
==========================================
  Files           9       11       +2     
  Lines        2125     2211      +86     
==========================================
+ Hits         2021     2094      +73     
- Misses        104      117      +13     
Flag Coverage Δ
unittests 94.70% <91.78%> (-0.40%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
trollflow2/launcher.py 86.11% <79.10%> (-3.14%) :arrow_down:
trollflow2/tests/test_launcher.py 97.86% <95.08%> (-0.80%) :arrow_down:
trollflow2/logging.py 100.00% <100.00%> (ø)
trollflow2/plugins/__init__.py 91.82% <100.00%> (ø)
trollflow2/tests/test_logging.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 582852b...1a63f48. Read the comment docs.