nasa / opera-sds-pcm

Observational Products for End-Users from Remote Sensing Analysis (OPERA)
Apache License 2.0
16 stars 12 forks source link

[Bug]: Failed download input products in GRQ_ES SLC catalog reported as TRUE in downloaded column field #446

Closed toandn-jpl closed 3 months ago

toandn-jpl commented 1 year ago

Checked for duplicates

Yes - I've already checked

Describe the bug

When I performed OPERA-SDSVnV-17_3: Timed test for retrieval of input products (L1_CSLC_S1) action to trigger l1_cslc_s1 download queries, I noticed failed downloaded jobs are still being marked as TRUE under downloaded field.

https://opera-int-mozart-pop1.jpl.nasa.gov/grq_es/

_index _type _id _score _source.id _source.granule_id _source.creation_timestamp _source.query_job_id _source.query_datetime _source.temporal_extent_beginning_datetime _source.revision_date _source.https_url _source.processing_mode _source.s3_url _source.download_datetime _source.download_job_id _source.downloaded _source.intersects_north_america
slc_catalog _doc S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip 0.015267471 S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3-SLC 2023-03-15T22:14:15.415131 d0f70977-f598-4197-b30a-ec9b26b94d18 2023-03-15T22:14:14.619226 2020-10-12T00:26:39+00:00 2023-02-24T23:04:37.161000+00:00 https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip forward s3://asf-ngap2w-p-s1-slc-7b420b89/S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip 2023-03-15T22:19:18.626680 9499d367-c0b0-4c2c-85e9-c3efc77f3b04 TRUE TRUE
[toandn@opera-int-mozart-pop1 ~]$ aws s3 cp s3://opera-int-triage-pop1/triaged_job-job-WF-slc_download-20230315T221415.498875Z_task-40ef7c7a-c1ca-4412-b166-b9edefaf1481/_stdout.txt .
download: s3://opera-int-triage-pop1/triaged_job-job-WF-slc_download-20230315T221415.498875Z_task-40ef7c7a-c1ca-4412-b166-b9edefaf1481/_stdout.txt to ./_stdout.txt

[toandn@opera-int-mozart-pop1 ~]$ cat _stdout.txt 
args: --endpoint=OPS --batch-ids S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip --start-date=1900-01-01T00:00:00Z --end-date=2023-03-15T22:14:14Z    --transfer-protocol=auto
##########################################
Running job to download ASFDAAC SLC data
Wed Mar 15 22:18:46 UTC 2023
python /home/ops/verdi/ops/opera-pcm/data_subscriber/daac_data_subscriber.py download -p ASF --endpoint=OPS --batch-ids S1A_IW_SLC__1SDV_20201012T002639_20201012T002707_034756_040CB8_B3A3.zip --start-date=1900-01-01T00:00:00Z --end-date=2023-03-15T22:14:14Z    --transfer-protocol=auto 2>&1
Failed to run daac_data_subscriber.py
Wed Mar 15 22:19:41 UTC 2023

What did you expect?

failed downloaded jobs should be marked as FALSE under downloaded field.

Reproducible steps

1. Run multiple slc download jobs
2. Check for any failed jobs via Figaro
3. Check https://opera-int-mozart-pop1.jpl.nasa.gov/grq_es/ slc_catalog versus grq_1_l1_s1_slc.

Environment

https://opera-int-mozart-pop1.jpl.nasa.gov/grq_es/
hhlee445 commented 3 months ago

Checked with latest version of PCM and don't see this error.