DISP-S1 PGE Workflow Compressed CSLC Support

[collinss-jpl]

Purpose

• This branch incorporates usage of the the list of Compressed CSLC products provided to a DISP-S1 SCIFLO job. In addition to including the CCSLCs in the RunConfig for a DISP-S1 job, this branch also adds the following:
  • CSLC download job now obtains the Ionosphere file(s) corresponding to the reference date of the CCSLCs to be used.
  • Fixed a bug where the incorrect S3 URLs where assigned to the ES metadata for Compressed CSLC products
  • The DISP-S1 workflow now utilizes the save_compressed_slcs flag provided to the job to assign the same field within the RunConfig
  • Simulation mode for DISP-S1 now generates a realistic number of Compressed CSLC products (27), and the sample filenames have been updated to be more realistic to the actual workflow.

Issues

• Resolves #911

Testing

• Branch has been tested on a dev cluster using DISP-S1 in both real and simulation mode. To test usage of the Compressed CSLCs and Ionosphere file staging, the following commands can be used from mozart: python ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_CSLC-S1_V1 --start-date=2016-07-05T00:00:00Z --end-date=2016-07-30T00:00:00Z --frame-id=10859 --use-temporal --chunk-size=1 --job-queue=opera-job_worker-cslc_data_download --processing-mode=historical --k=2 --m=1 This first job will not utilize any CCSLCs, but will create some to be used by the next job. Let the first job run to completion, then run: python ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_CSLC-S1_V1 --start-date=2016-08-22T00:00:00Z --end-date=2016-09-16T00:00:00Z --frame-id=10859 --use-temporal --chunk-size=1 --job-queue=opera-job_worker-cslc_data_download --processing-mode=historical --k=2 --m=2 This job will utilize the CCSLCs created by the first job.

[philipjyoon]

Looks good to me. Note that save_compressed_slcs was a misspelling that was fixed recently. Looking at the code changes here this code uses the correct flag save_compressed_cslc so all is good. You may want to just update that in your comments.

[hhlee445]

DISP-S1 failed with a following command.

python ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_CSLC-S1_V1 --chunk-size=1 --k=4 --job-queue=opera-job_worker-cslc_data_download --processing-mode=reprocessing --native-id=OPERA_L2_CSLC-S1_T122-260017-IW1_20231120T011410Z_20240516T230101Z_S1A_VV_v1.1 --m=1

http://opera-dev-triage-fwd-hyunlee.s3-website-us-west-2.amazonaws.com/triaged_job-job-WF-SCIFLO_L3_DISP_S1-frame-32502-latest_acq_index-2628-20240719T011657.639137Z_task-5da5517d-f586-41ec-95ce-4f1b0fe17d29

[collinss-jpl]

DISP-S1 failed with a following command.

python ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_CSLC-S1_V1 --chunk-size=1 --k=4 --job-queue=opera-job_worker-cslc_data_download --processing-mode=reprocessing --native-id=OPERA_L2_CSLC-S1_T122-260017-IW1_20231120T011410Z_20240516T230101Z_S1A_VV_v1.1 --m=1

http://opera-dev-triage-fwd-hyunlee.s3-website-us-west-2.amazonaws.com/triaged_job-job-WF-SCIFLO_L3_DISP_S1-frame-32502-latest_acq_index-2628-20240719T011657.639137Z_task-5da5517d-f586-41ec-95ce-4f1b0fe17d29

Hi @hhlee445, I've pushed a commit that fixes this issue. I've also tested this command on my dev cluster to verify that the job now runs to completion. Let me know if you find anything else.