nasa / opera-sds-pcm

Observational Products for End-Users from Remote Sensing Analysis (OPERA)
Apache License 2.0
16 stars 12 forks source link

Issue 939: adjust hysds payload for proper deduping #940

Closed chrisjrd closed 1 month ago

chrisjrd commented 1 month ago

Purpose

Issues

Refs #939

Testing

riverma commented 1 month ago

@chrisjrd @hhlee445 - can we add this as an integration test case in https://github.com/nasa/opera-sds-pcm/blob/develop/tests/integration/test_integration.py or equivalent? In general - it's good software engineering practice to ensure bugs that are found are added to automated test cases to prevent regression in the future.

philipjyoon commented 1 month ago

@chrisjrd have you verified that the corresponding SCIFLO triggering is also not deduped? Or does that also need to be changed up a bit?

hhlee445 commented 1 month ago

Verified that the second job didn't get deduped and generated more DSWx-S1 products by later job

  1. 'MS_147_4$168', 'e5540925-ff88-4c0c-8182-b6ea374dd1df' was triggered with

python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_RTC-S1_V1 --job-queue=opera-job_worker-rtc_data_download --chunk-size 1 --processing-mode=forward -s 2024-07-19T00:15:19Z -e 2024-07-19T01:15:19Z

MS_147_4 - 27 RTCs were used

  1. 'MS_147_4$168', '29a96e5b-ffc7-409b-9e9c-1c878a8494b7' was triggered with

python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -c OPERA_L2_RTC-S1_V1 --job-queue=opera-job_worker-rtc_data_download --chunk-size 1 --processing-mode=forward -s 2024-07-19T07:15:19Z -e 2024-07-19T08:15:19Z

MS_147_4 - 41 RTCs were used

Screenshot 2024-08-07 at 2 30 55 PM