search

podaac / l2ss-py

Level 2 subsetter with Harmony integration
https://podaac.github.io/l2ss-py/
Apache License 2.0
11 stars 11 forks source link

SWOT collections failing temporal subsetting #189

Open jamesfwood opened 1 year ago

jamesfwood commented 1 year ago

The SWOT collections are failing temporal subsetting.

This simulated harmony request works, but the real collections fail.

Simulated (Working): https://harmony.earthdata.nasa.gov/C2152044763-POCLOUD/ogc-api-coverages/1.0.0/collections/ssh_karin%2Cssha_karin%2Cssh_karin_2%2Cssha_karin_2%2Ctime%2Ctime_tai%2Clongitude%2Clatitude/coverage/rangeset?subset=lon(-125.57812%3A-124.73438)&subset=lat(34.84931%3A35.83342)&subset=time(%222012-01-01T00%3A00%3A00Z%22%3A%222012-03-31T23%3A59%3A59Z%22)&format=application%2Fnetcdf

Make all the real collections work also:

SWOT_L2_LR_SSH_BASIC_1.0 SWOT_L2_LR_SSH_EXPERT_1.0 SWOT_L2_LR_SSH_UNSMOOTHED_1.0 SWOT_L2_LR_SSH_WINDWAVE_1.0 SWOT_L2_NALT_IGDR_SSHA_1.0 SWOT_L2_NALT_OGDR_SSHA_1.0

jamesfwood commented 1 year ago

References here: https://wiki.jpl.nasa.gov/display/PD/Data+Subscriber+1.15.0+UAT

sliu008 commented 12 months ago

https://github.com/podaac/l2ss-py/pull/190

jamesfwood commented 11 months ago

Test in Data Subscriber when ready.

jamesfwood commented 11 months ago

@davidcolemanjpl l2ss-py was updated in harmony in UAT only. Please try to test this in UAT only. You may need help from @jjmcnelis to setup the test case. Thanks!

davidcolemanjpl commented 11 months ago

@jamesfwood I'l review, but will need more details regarding this validation:

E.g., Inputs / Expected results

Version(s) under test (l2ss-py) -- 'l2ss-py:2.6.0rc11'

jamesfwood commented 11 months ago

@davidcolemanjpl Please test using the Data Subscriber. It's only deployed to UAT so far, so test in UAT. I think you can get a test plan from either @jjmcnelis or @skorper. This was an important fix for them.
Let me know if you need more help. Thanks!

jamesfwood commented 10 months ago

@davidcolemanjpl you can also test this using the harmony call directly:

https://harmony.uat.earthdata.nasa.gov/C1256507988-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-18T18:00:00%2B00:00":"2023-04-20T07:00:00%2B00:00")&skipPreview=true

Something like this, but update the subset lon and lat and tweak the time a bit.

jamesfwood commented 10 months ago

@davidcolemanjpl here's another one that should work. Just tweak the times.

https://harmony.uat.earthdata.nasa.gov/C1256507988-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=time("2023-04-19T02:00:00%2B00:00":"2023-04-19T04:00:00%2B00:00")&skipPreview=true

Hopefully you can test this soon so we can do a release this week. Thanks!

davidcolemanjpl commented 10 months ago

l2ss-py:2.6.0rc14; podaac-data-subscriber v1.15.0a3 UAT

@jamesfwood Thanks, ...the harmony jobID seems to trigger as expected when using call for the SWOT_L2_LR_SSH_BASIC_1.0 collection.

Are there example, valid harmony calls for exercising other SWOT sub-collections under test?: SWOT_L2_LR_SSH_EXPERT_1.0 SWOT_L2_LR_SSH_UNSMOOTHED_1.0 SWOT_L2_LR_SSH_WINDWAVE_1.0 SWOT_L2_NALT_IGDR_SSHA_1.0 SWOT_L2_NALT_OGDR_SSHA_1.0

note: When attempting to execute a harmony call directly using the SWOT_L2_LR_SSH_EXPERT_1.0 collection (C1256122852-POCLOUD), the jobID hung at the running status See jobID : efe48442-72ee-454f-8e13-93c3202a726e
https://harmony.uat.earthdata.nasa.gov/workflow-ui/efe48442-72ee-454f-8e13-93c3202a726e

See jobID: 6d36b26c-be0e-4423-840c-a005fa0dc903. - FAILED WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error

https://harmony.uat.earthdata.nasa.gov/C1256122852-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-17T18:00:00%2B00:00":"2023-04-19T07:00:00%2B00:00")&skipPreview=true

See jobID: 02e0e68f-bf23-4960-b0e0-4ce0687672da "The job failed with 2 errors. See the errors field for more details"

note2: The L2SS-Py functionality seems to work okay when sub-setting SWOT_L2_LR_SSH_EXPERT_1.0 collection nominally (via HiTIDE) : Can see the successful jobID 14682326-2b58-4b85-9d1e-097c3a928653 https://harmony.uat.earthdata.nasa.gov/jobs/14682326-2b58-4b85-9d1e-097c3a928653

podaac-data-subscriber v1.15.0a3: - anomaly _SWOT_L2_LR_SSH_BASIC1.0

% podaac-data-downloader -c SWOT_L2_LR_SSH_BASIC_1.0 -d ./data --start-date 2023-04-17T18:00:00Z --end-date 2023-04-21T07:00:00Z --subset -b="-125.57812,34.84931,-124.73438,35.83342" [2023-10-04 12:55:57,631] {podaac_data_downloader.py:382} ERROR - Uncaught exception occurred during execution. Traceback (most recent call last): File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_data_downloader.py", line 380, in main run() File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_data_downloader.py", line 142, in run collection_id = pa.get_cmr_collection_id( File "/Users/colemand/work/env/lib/python3.10/site-packages/subscriber/podaac_access.py", line 574, in get_cmr_collection_id raise ValueError(f'No collections found in CMR for {collection_short_name}/{provider}') ValueError: No collections found in CMR for SWOT_L2_LR_SSH_BASIC_1.0/POCLOUD

(I'll keep investigating this)

davidcolemanjpl commented 10 months ago

l2ss-py:2.6.0rc14; podaac-data-subscriber v1.15.0a3 UAT SWOT_L2_LR_SSH_WINDWAVE_1.0 (C1256507990-POCLOUD)

See jobID: c3b76190-f155-4f5e-aa8a-bc2d5da6270e - FAILED harmony call : https://harmony.uat.earthdata.nasa.gov/C1256507990-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-04-18T18:00:00%2B00:00":"2023-04-20T07:00:00%2B00:00")&skipPreview=true

"The job failed with 3 errors. See the errors field for more details"

https://harmony.uat.earthdata.nasa.gov/jobs/c3b76190-f155-4f5e-aa8a-bc2d5da6270e

podaac-data-subscriber v1.15.0a3: - anomaly: _"ValueError: No collections found in CMR for SWOT_L2_LR_SSH_WINDWAVE1.0/POCLOUD"

note: The L2SS-Py functionality seems to work okay when subsetting SWOT_L2_LR_SSH_WINDWAVE collection nominally (via HiTIDE)

davidcolemanjpl commented 10 months ago

l2ss-py:2.6.0rc14; podaac-data-subscriber v1.15.0a3

UAT

SWOT_L2_LR_SSH_BASIC_1.0: Data subscriber issue: ValueError: No collections found in CMR for C1256507988-POCLOUD/POCLOUD

SWOT_L2_LR_SSH_EXPERT_1.0: Data subscriber issue: _ValueError: No collections found in CMR for SWOT_L2_LR_SSH_EXPERT1.0/POCLOUD

When using Harmony call directly, the subset failed for SWOT_L2_LR_SSH_EXPERT1.0 (see Harmony jobID: 178ea40f-643a-4d78-8ea3-c8e8c49ac038) "WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error"_ https://harmony.uat.earthdata.nasa.gov/jobs/178ea40f-643a-4d78-8ea3-c8e8c49ac038

SWOT_L2_LR_SSH_UNSMOOTHED_1.0: Data subscriber issue: _ValueError: No collections found in CMR for SWOT_L2_LR_SSH_UNSMOOTHED1.0/POCLOUD

SWOT_L2_LR_SSH_WINDWAVE_1.0: Data subscriber issue: _ValueError: No collections found in CMR for SWOT_L2_LR_SSH_WINDWAVE1.0/POCLOUD

When using Harmony call Directly, the subset FAILED for SWOT_L2_LR_SSH_WINDWAVE1.0 (see Failed jobID: c3b76190-f155-4f5e-aa8a-bc2d5da6270e) "WorkItem failed: podaac/l2ss-py:2.6.0rc14: Service request failed with an unknown error"_ https://harmony.uat.earthdata.nasa.gov/jobs/c3b76190-f155-4f5e-aa8a-bc2d5da6270e

SWOT_L2_NALT_IGDR_SSHA_1.0: l2ss-py:2.6.0rc14 - OKAY - PASS podaac-data-subscriber v1.15.0a3 - OKAY - PASS

SWOT_L2_NALT_OGDR_SSHA_1.0: l2ss-py:2.6.0rc14 - OKAY - PASS podaac-data-subscriber v1.15.0a3 - OKAY - PASS

can also see related podaac-data-subscriber ticket: PODAAC-5859

jamesfwood commented 10 months ago

@davidcolemanjpl how do you run data subscriber with UAT sources? collection ids, using uat services, etc.

davidcolemanjpl commented 10 months ago

Correct, utilized the respective Data Subscriber UAT Install Instructions

jamesfwood commented 10 months ago

@davidcolemanjpl We deployed a new l2ss-py to OPS. Can you please test this again in OPS?

jamesfwood commented 10 months ago

Harmony may be down right now, so please test when it's back running again. Test here: https://harmony.earthdata.nasa.gov/workflow-ui

davidcolemanjpl commented 10 months ago

HITIDE UI OPS V4.16.1 L2SS-PY:v2.6.0 CONCISE: v0.6.1

@jamesfwood

Related SWOT L2 collections currently displaying in HiTIDE UI (OPS): SWOT_L2_NALT_IGDR_SSHA_1.0 -- (C2628598397-POCLOUD) -- PASSED

https://harmony.earthdata.nasa.gov/jobs/5945fd10-8785-4e56-9d7f-032beccc6623 (L2SS-py PASS via HiTIDE UI) https://harmony.earthdata.nasa.gov/jobs/a6feb324-8b34-4900-b776-f7eebc1a0ae6 (L2SS-py PASS using Harmony call)

SWOT_L2_NALT_OGDR_SSHA_1.0 -- (C2628593693-POCLOUD) -- PASSED
https://harmony.earthdata.nasa.gov/jobs/5df9ba70-5243-456a-b7b2-d00a1ee5adcd (L2SS-py PASS via HiTIDE UI) https://harmony.earthdata.nasa.gov/jobs/413e36e1-fad7-493d-a7c6-7f988e8dbb3d (L2SS-py PASS using Harmony call)

note: the following SWOT L2 LR collections have not been implemented thus far in HiTIDE OPS UI and/or EARTHDATASEARCH: 'BLOCKED' **SWOT_L2_LR_SSH_BASIC_1.0

SWOT_L2_LR_SSH_EXPERT_1.0

SWOT_L2_LR_SSH_UNSMOOTHED_1.0 SWOT_L2_LR_SSH_WINDWAVE_1.0**

See related issue for SWOT collections: https://github.com/podaac/hitide/issues/13

UPDATE: 10/18: I was able to create / utilze Harmony call in OPS for SWOT_L2_LR_SSH_EXPERT_1.0 granules : (L2SS-py - Conditional PASS using Harmony call): (_SWOT Simulated Level-2 KaRIn SSH from MITgcm ECCO LLC4320 for Science Version 1 collection / SWOT_SIMULATED_L2_KARIN_SSH_ECCO_LLC4320_SCIENCEV1 / C2152044763-POCLOUD)

https://harmony.earthdata.nasa.gov/C2152044763-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2012-03-01T18:00:00%2B00:00":"2023-04-21T07:00:00%2B00:00")&skipPreview=true

See Harmony jobID : a98c7b20-8634-4b02-a867-f3bee4cc4c45

"The job has completed with errors. See the errors field for more details" "message":"WorkItem failed: podaac/l2ss-py:2.6.0: Service request failed with an unknown error"

https://harmony.earthdata.nasa.gov/jobs/a98c7b20-8634-4b02-a867-f3bee4cc4c45