R3 PGE v3.0.0-RC.2.1 Integration

[collinss-jpl]

Purpose

• This branch integrates the v3.0.0-RC.2.1 deliveries of the R3 PGEs (DSWx-S1 and DISP-S1) into OPERA PCM.
• v3.0.0-RC.2.1 of the DSWx-S1 PGE wraps the CalVal SAS point delivery v4.2, which fixes an issue where NaN values could be assigned into the product metadata, and subsequently the iso.xml product
• v3.0.0-RC.2.1 of the DISP-S1 PGE wraps the Gamma SAS point delivery v4.1, which addresses some issues related to usage of input CSLC-S1 products with naming conventions, and adds naming conventions for Compressed CSLC output products

Proposed Changes

• [ADD] Regular expressions for Compressed CSLC products have been updated to use latest official file name conventions
• [ADD] Regular expressions for DISP-S1 Browse product have been updated to include "_BROWSE" suffix
• [ADD] This branch adds support for staging the CSLC static layer and Water mask ancillary inputs
• [CHANGE] This branch fixes a bug in product2dataset.py where attempting to read a dummy iso.xml file created during a simulated PGE run resulted in a crash
• [REMOVE] Support for inclusion of Ionosphere ancillary inputs has been temporarily disabled, as there are issues in the SAS when running atmospheric corrections with CSLC inputs that have file name conventions applied

Issues

• Resolves #775
• Resolves #742

Testing

• This branch was tested on a dev cluster, running both PGE types in both real and simulation mode.
• To test DSWx-S1 triggering, the following command was used: python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query --collection-shortname=OPERA_L2_RTC-S1_V1 --endpoint=OPS --release-version=issue_775_r3_pge_v3.0.0-rc.2.1_integration --job-queue=opera-job_worker-rtc_data_download --chunk-size=1 --transfer-protocol=auto --native-id=OPERA_L2_RTC-S1_T024-050544-IW1_20240312T084422Z_20240409T135838Z_S1A_30_v1.0
• To test DISP-S1 triggering, the following command was used: python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query --collection-shortname=OPERA_L2_CSLC-S1_V1 --endpoint=OPS --k=4 --release-version=issue_775_r3_pge_v3.0.0-rc.2.1_integration --job-queue=opera-job_worker-cslc_data_download --chunk-size=1 --processing-mode=reprocessing --transfer-protocol=auto --native-id=OPERA_L2_CSLC-S1_T042-088905-IW1_20240301T140753Z_20240302T185554Z_S1A_VV_v1.0

[hhlee445]

Tested with

1. https://github.com/nasa/opera-sds/blob/riverma-edge-case-table-fix/processing_request_datasets/edge_cases/rtc_antimeridian_complete_20231004_thru_20240311.txt
2. https://github.com/nasa/opera-sds/blob/riverma-edge-case-table-fix/processing_request_datasets/edge_cases/rtc_antimeridian_complete_20231004_thru_20240311_landonly.txt
3. https://github.com/nasa/opera-sds/blob/riverma-edge-case-table-fix/processing_request_datasets/edge_cases/rtc_high_latitude_complete_20231004_thru_20240311.txt
4. https://github.com/nasa/opera-sds/blob/riverma-edge-case-table-fix/processing_request_datasets/edge_cases/rtc_high_latitude_complete_20231004_thru_20240311_landonly.txt
5. https://github.com/nasa/opera-sds/blob/riverma-edge-case-table-fix/processing_request_datasets/edge_cases/dswx-s1_orbit_boundary_test.txt

171 DSWx-S1 jobs were executed and completed successfully.

9 DISP-S1 sciflo jobs were executed with cslc timer and 8 jobs failed with

Starting SAS execution for DispS1Executor Traceback (most recent call last): File "/home/mamba/opera/scripts/pge_main.py", line 190, in pge_main() File "/home/mamba/opera/scripts/pge_main.py", line 186, in pge_main pge_start(run_config_filename) File "/home/mamba/opera/scripts/pge_main.py", line 160, in pge_start pge.run() File "/home/mamba/opera/pge/base/base_pge.py", line 760, in run self.run_sas_executable(kwargs) File "/home/mamba/opera/pge/base/base_pge.py", line 739, in run_sas_executable elapsed_time = time_and_execute( ^^^^^^^^^^^^^^^^^ File "/home/mamba/opera/util/run_utils.py", line 216, in time_and_execute logger.critical(module_name, ErrorCode.SAS_PROGRAM_FAILED, error_msg) File "/home/mamba/opera/util/logger.py", line 407, in critical raise RuntimeError(description) RuntimeError: Command "['/opt/conda/bin/disp-s1', 'run', '/home/mamba/scratch_dir/RunConfig_sas.yaml']" failed with exit code 1, traceback: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/disp_s1/cli/run.py", line 19, in run pge_runconfig = RunConfig.from_yaml(config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/dolphin/workflows/config/_yaml_model.py", line 87, in from_yaml return cls(**data) ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/pydantic/main.py", line 171, in init self.__pydantic_validator__.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 1 validation error for RunConfig dynamic_ancillary_file_group.static_layers_files Input should be a valid list [type=list_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.6/v/list_type

1 job failed with

Traceback (most recent call last): File "/home/mamba/opera/scripts/pge_main.py", line 190, in pge_main() File "/home/mamba/opera/scripts/pge_main.py", line 186, in pge_main pge_start(run_config_filename) File "/home/mamba/opera/scripts/pge_main.py", line 160, in pge_start pge.run() File "/home/mamba/opera/pge/base/base_pge.py", line 760, in run self.run_sas_executable(kwargs) File "/home/mamba/opera/pge/base/base_pge.py", line 739, in run_sas_executable elapsed_time = time_and_execute( ^^^^^^^^^^^^^^^^^ File "/home/mamba/opera/util/run_utils.py", line 216, in time_and_execute logger.critical(module_name, ErrorCode.SAS_PROGRAM_FAILED, error_msg) File "/home/mamba/opera/util/logger.py", line 407, in critical raise RuntimeError(description) RuntimeError: Command "['/opt/conda/bin/disp-s1', 'run', '/home/mamba/scratch_dir/RunConfig_sas.yaml']" failed with exit code 1, traceback: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/disp_s1/cli/run.py", line 19, in run pge_runconfig = RunConfig.from_yaml(config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/dolphin/workflows/config/_yaml_model.py", line 87, in from_yaml return cls(**data) ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/pydantic/main.py", line 171, in init self.__pydantic_validator__.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 1 validation error for RunConfig dynamic_ancillary_file_group.static_layers_files Input should be a valid list [type=list_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.6/v/list_type

Running preprocessor for DispS1PreProcessorMixin Starting SAS execution for DispS1Executor