Closed dostuffthatmatters closed 1 month ago
Currently, the retrieval queue will consider any day where the directory of interferograms exist: https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/dispatching/retrieval_queue.py#L56
However, on some days, these directories are empty (or only contain subdirectories). Checking them all with os.path.isdir is inefficient, hence we should do something in a subprocess listing everything that is a file and only include that day if there is at least one file. We can also use the regex which is used in the final filtering (https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/session/move_ifg_files.py#L31-L35).
os.path.isdir
Will not be worked on because this would make the retrieval queue significantly slower (2-3 orders of magnitude slower).
Currently, the retrieval queue will consider any day where the directory of interferograms exist: https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/dispatching/retrieval_queue.py#L56
However, on some days, these directories are empty (or only contain subdirectories). Checking them all with
os.path.isdir
is inefficient, hence we should do something in a subprocess listing everything that is a file and only include that day if there is at least one file. We can also use the regex which is used in the final filtering (https://github.com/tum-esm/em27-retrieval-pipeline/blob/1a744f1a05ea516d92ff3f7224e8612e990e9da6/src/retrieval/session/move_ifg_files.py#L31-L35).