DSWx-S1 PGE RC2.0 Integration

[collinss-jpl]

Purpose

• This branch integrates v3.0.0-rc.2.0 of the DSWx-S1 PGE, which incorporates the CalVal delivery of the corresponding SAS

Proposed Changes

• The designated instance type for DSWx-S1 workers is now set to c5.4xlarge
• References to the MGRS tile collection database have been updated to use the newest version with the CalVal delivery. All usages now reference a single location in S3
• The DSWx-S1 workflow now assigns the number of available CPU to 1 less than the total (15 for c5.4xlarge)
• This branch also updates the opera_pge_wrapper.py script to create a distinct scratch directory and mount it into the docker container. This directory will also be collected when triaging a failed job. All existing PGE job-specs have been updated to support this change.

Issues

• Resolves #755

Testing

• Branch was tested on a dev cluster by submitting a single DSWx-S1 job with the following command:

  python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query --collection-shortname=OPERA_L2_RTC-S1_V1 --endpoint=OPS --start-date=2024-01-29T00:00:00Z --end-date=2024-01-29T00:15:00Z --release-version=issue_755_dswx_s1_rc2.0_integration --job-queue=opera-job_worker-rtc_data_download --chunk-size=1 --max-revision=1000 --temporal-start-date=2023-12-28T00:00:00Z --transfer-protocol=auto

  Both simulation mode and real mode were tested.

• Testing is also ongoing with the SDS in forward processing mode

[hhlee445]

DSWx-S1 jobs finished about 2000 secs with c5.4xlarge instance type. 2 failed jobs out of 55 DSWx-S1 sciflo jobs.