Add a way to populate segment gatherer with pre-existing files

[pnuu]

This PR adds a config option that makes it possible to populate the segment gatherer with pre-existing files. The check is triggered only for the first received message after segment_gatherer.py is started. Only "file" messages and local filesystem are supported.

As use case, consider the segmented MSG SEVIRI data from Eumetcast. If a restart is required due to configuration change or upgrade, currently the restart needs to be done either during the short gap between two timeslots, re-send the messages from new files, or potentially lose all files that arrived before the restart, or even the full set if prologue or epilogue files are missed.

With this PR, the first incoming message triggers (if check_existing_files_after_start is set to true in config) a check for all the files in the same directory where the 'uri' in the message points to. The matching files will be added to the newly created slot.

[codecov[bot]]

Codecov Report

Merging #84 (668c3ef) into main (6eeb3c1) will increase coverage by 5.44%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #84      +/-   ##
==========================================
+ Coverage   72.83%   78.27%   +5.44%     
==========================================
  Files          13       14       +1     
  Lines        2466     2582     +116     
==========================================
+ Hits         1796     2021     +225     
+ Misses        670      561     -109     

Impacted Files	Coverage Δ
pytroll_collectors/segments.py	89.93% <100.00%> (+0.62%)	:arrow_up:
pytroll_collectors/tests/test_segments.py	99.19% <100.00%> (+0.04%)	:arrow_up:
pytroll_collectors/tests/test_region_collector.py	92.72% <0.00%> (ø)
pytroll_collectors/region_collector.py	78.35% <0.00%> (+78.35%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6eeb3c1...668c3ef. Read the comment docs.

[ghost]

DeepCode's analysis on #668c3e found:

• :information_source: 1 minor issue. :point_down:

  Top issues

Description	Example fixes
assertTrue cannot provide an informative message about the values in the comparison _ if it fails Occurrences: test_segments.py:111 test_segments.py:122 test_segments.py:123	:wrench: Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot