search

nasa / opera-sds-pcm

Observational Products for End-Users from Remote Sensing Analysis (OPERA)
Apache License 2.0
16 stars 12 forks source link

Data Subscriber Product Catalog Class Refactor #893

Closed collinss-jpl closed 2 months ago

collinss-jpl commented 3 months ago

Purpose

This branch refactors the classes used to catalog product information to Elasticsearch into a single coherent hierarchy. All catalog classes now inherit from a single ProductCatalog class, and all redundant or unused code has been removed. Additionally, a unit test suite for ProductCatalog and all its inheritors has also been added.

Testing

philipjyoon commented 2 months ago

Perused through and the changes generally look good. Have not had time to look deeper, however.

We are looking to do a release in the next few days. I recommend that we don't merge this PR until that release is completed. It would be too risky and we may not have enough time to review all of this prior to that time.

collinss-jpl commented 2 months ago

@hhlee445 I have rebased this branch against the latest develop branch that includes the updates for 3.1.0-rc.3.0. Feel free to merge this branch at your convenience.

hhlee445 commented 2 months ago

@collinss-jpl Getting a warning with a command and didn't trigger any query/download jobs

python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -s 2022-01-17T00:00:00Z -e 2022-01-17T00:05:00Z -c HLSS30 --job-queue=opera-job_worker-hls_data_download --chunk-size=2 --use-temporal

[2024-07-10 18:34:15,302: INFO/run] END /export/home/hysdsops/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py:56: RuntimeWarning: coroutine 'run_query' was never awaited run(sys.argv) RuntimeWarning: Enable tracemalloc to get the object allocation traceback

collinss-jpl commented 2 months ago

python3 ~/mozart/ops/opera-pcm/data_subscriber/daac_data_subscriber.py query -s 2022-01-17T00:00:00Z -e 2022-01-17T00:05:00Z -c HLSS30 --job-queue=opera-job_worker-hls_data_download --chunk-size=2 --use-temporal

Hi @hhlee445, this error should now be fixed. I have also retested with subscriber commands for all other product lines (SLC, RTC, CSLC) that we query for and they seem to be working on my end.