tum-esm / em27-retrieval-pipeline

automated EM27 data processing
https://em27-retrieval-pipeline.netlify.app
GNU General Public License v3.0
6 stars 1 forks source link

Do not try to process days without interferograms #88

Closed dostuffthatmatters closed 1 month ago

dostuffthatmatters commented 9 months ago

Currently, the retrieval queue will consider any day where the directory of interferograms exist: https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/dispatching/retrieval_queue.py#L56

However, on some days, these directories are empty (or only contain subdirectories). Checking them all with os.path.isdir is inefficient, hence we should do something in a subprocess listing everything that is a file and only include that day if there is at least one file. We can also use the regex which is used in the final filtering (https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/session/move_ifg_files.py#L31-L35).

dostuffthatmatters commented 1 month ago

Will not be worked on because this would make the retrieval queue significantly slower (2-3 orders of magnitude slower).