add pipeline status monitor

nicHoch commented 10 months ago

Status logger and notification script for the pipeline.

SetUp via cron.
Query the number of open files still to process. Logs that number into a status file
and checks if the the number is constantly equal or increasing.

Sends an notification via mail if a possible pipeline stuck is detected.

nicHoch commented 9 months ago

@samaloney "unfortunately" we missed a stuck case on this weekend. i waited until i get the mail but forgot that i changed to another branch in me dev environment ... but if you review it i can merge it into master soon

nicHoch commented 9 months ago

all our entry-points / mains clis for regular use are in this processing folder. the script are more used as helpers on times ...

i will also add a entry point in the project setup

codecov-commenter commented 9 months ago

Codecov Report

Attention: 61 lines in your changes are missing coverage. Please review.

Comparison is base (a801e54) 77.79% compared to head (3b00fa6) 77.12%.

Files	Patch %	Lines
stixcore/processing/pipeline_monitor.py	0.00%	59 Missing :warning:
stixcore/processing/pipeline_status.py	33.33%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #372 +/- ## ========================================== - Coverage 77.79% 77.12% -0.68% ========================================== Files 65 66 +1 Lines 6777 6836 +59 ========================================== Hits 5272 5272 - Misses 1505 1564 +59 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

i4Ds / STIXCore

add pipeline status monitor #372

Codecov Report