search

HEPCloud / decisionengine

HEPCloud Decision Engine framework
Apache License 2.0
6 stars 25 forks source link

Fix message explosion #632

Closed knoepfel closed 2 years ago

knoepfel commented 2 years ago

This PR ensures that the queue used by a source to publish data is shared with any channels that require that data. This PR also introduces the --queue-status program option to the DE client:

$ de-client --queue-status

Source name      Queue name                                            Unconsumed messages
---------------  ----------------------------------------------------  ---------------------
first_source     first_source-EFBF49B8-1186-4C08-AA53-D52E6AB1A4E4     None
last_source_A    last_source_A-B56713E6-5B78-44DF-9FFE-E3A0BC93ACB0    None
last_source_B    last_source_B-57D783C3-2663-4B5A-81F7-8B7934CDD490    None
second_source_A  second_source_A-546B7A71-FF42-40B7-A790-92C9A67CD38E  None
second_source_B  second_source_B-AB584B12-3A7D-48FA-B6E0-380985C72B1C  None
second_source_C  second_source_C-1B5B60AB-28AB-4859-AD72-2D30B594FBD5  None
codecov[bot] commented 2 years ago

Codecov Report

Merging #632 (858f5df) into master (97aed84) will decrease coverage by 0.13%. The diff coverage is 100.00%.

:exclamation: Current head 858f5df differs from pull request most recent head 6d58dbf. Consider uploading reports for the commit 6d58dbf to get more accurate results

@@            Coverage Diff             @@
##           master     #632      +/-   ##
==========================================
- Coverage   97.35%   97.21%   -0.14%     
==========================================
  Files          47       47              
  Lines        2838     2840       +2     
  Branches      449      450       +1     
==========================================
- Hits         2763     2761       -2     
- Misses         50       52       +2     
- Partials       25       27       +2
Flag Coverage Δ
python-3.10 97.00% <100.00%> (+0.03%) :arrow_up:
python-3.6 96.74% <100.00%> (-0.18%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../decisionengine/framework/engine/DecisionEngine.py 91.82% <100.00%> (-0.20%) :arrow_down:
...c/decisionengine/framework/engine/SourceWorkers.py 100.00% <100.00%> (ø)
...ecisionengine/framework/taskmanager/TaskManager.py 97.44% <100.00%> (+0.01%) :arrow_up:
.../decisionengine/framework/engine/ChannelWorkers.py 88.15% <0.00%> (-2.64%) :arrow_down:
src/decisionengine/framework/dataspace/maintain.py 99.04% <0.00%> (-0.96%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2ce5ccb...6d58dbf. Read the comment docs.