This branch represents an initial integration of support for ECMWF (Troposphere) file usage with the DISP-S1 PGE workflow. A precondition function in the Chimera workflow for DISP-S1 will now derive the set of ECMWF files for use with a DISP-S1 job based on the CSLC products (compressed and standard) staged for the job. Note that further refinements may be required to this branch once a sufficient number of subset/compressed ECMWF files are available for further testing.
A flag to control the behavior of the precondition function has also been added to settings.yaml, named STRICT_ANCILLARY_USAGE. If this flag is set to True, then any expected ECMWF files that are not available in S3 will result in a failed DISP-S1 job. If set to False, then any missing files will result in a DISP-S1 job with no troposphere files provided. This was added to allow testing to continue while the back catalog of subset and compressed ECMWF files is still being populated. When in production, the STRICT_ANICLLARY_USAGE flag should be set to True.
Lastly, a new utility module, ecmwf_utils.py, has been added to collect the functions used to derive ECMWF S3 paths, and check for their availability. One function in particular, find_ecmwf_for_datetime(), has been added for use with the CSLC file staging logic to ensure that all ECMWF files are available before triggering a DISP-S1 job.
Issues
Resolves #967
Testing
Unit tests have been added for both the new ecmwf_utils.py module, as well as the new precondition function used to assign ECMWF files to the DISP-S1 RunConfig.
This branch was also tested on a dev cluster using both simulation and real mode for DISP-S1, as well as with the STRICT_ANCILLARY_USAGE flag enabled and disabled.
Purpose
STRICT_ANCILLARY_USAGE. If this flag is set to True, then any expected ECMWF files that are not available in S3 will result in a failed DISP-S1 job. If set to False, then any missing files will result in a DISP-S1 job with no troposphere files provided. This was added to allow testing to continue while the back catalog of subset and compressed ECMWF files is still being populated. When in production, theSTRICT_ANICLLARY_USAGEflag should be set to True.ecmwf_utils.py, has been added to collect the functions used to derive ECMWF S3 paths, and check for their availability. One function in particular,find_ecmwf_for_datetime(), has been added for use with the CSLC file staging logic to ensure that all ECMWF files are available before triggering a DISP-S1 job.Issues
Testing
ecmwf_utils.pymodule, as well as the new precondition function used to assign ECMWF files to the DISP-S1 RunConfig.STRICT_ANCILLARY_USAGEflag enabled and disabled.