search

podaac / l2ss-py

Level 2 subsetter with Harmony integration
https://podaac.github.io/l2ss-py/
Apache License 2.0
11 stars 11 forks source link

Error running l2ss-py on SWOT_L2_LR_SSH_Expert_2.0 #266

Open skorper opened 1 month ago

skorper commented 1 month ago

The following Harmony request is failing:

https://harmony.earthdata.nasa.gov/C2799465497-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(30.3:38.1)&subset=lon(-140.4:-122.9)&subset=time("2024-03-24T00:00:00+00:00":"2024-04-01T00:00:00+00:00")&skipPreview=true&ignoreErrors=true

Error is WorkItem failed: podaac/l2ss-py:2.9.0: Service request failed with an unknown error

This issue was found using the data subscriber with this command:

podaac-data-downloader -c SWOT_L2_LR_SSH_Expert_2.0 -d ./data --start-date 2024-03-24T00:00:00Z --end-date 2024-04-01T00:00:00Z -b="-140.4,30.3,-122.9,38.1" --subset

Version: l2ss-py:2.9.0

sliu008 commented 1 month ago

Looks like l2ss is finding the wrong time variable, we have a fix for this in a upcoming l2ss release

jamesfwood commented 1 month ago

@davidcolemanjpl Please test this in UAT and OPS using the new l2ss-py 2.10.0. Thanks!

davidcolemanjpl commented 1 month ago

NOTE: the Listed API Harmony request noted in this ticket is erroneous, returns error:

https://harmony.earthdata.nasa.gov/C2799465497-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(30.3:38.1)&subset=lon(-140.4:-122.9)&subset=time("2024-03-24T00:00:00+00:00":"2024-04-01T00:00:00+00:00")&skipPreview=true&ignoreErrors=true

ERROR: { "code": "harmony.RequestValidationError", "description": "Error: query parameter \"subset\" subset dimension \"time\" has an invalid date time \"2024-03-24T00:00:00 00:00\"" }

davidcolemanjpl commented 1 month ago

Harmony OPS - v 0.0.1029 Data subscriber - v1.15.2 L2SS-Py - v2.10.0 Harmony UAT- v 0.0.1029 (l2ss-py:2.10.0rc8) (Related to https://github.com/podaac/hitide/issues/51)

SWOT_L2_LR_SSH_EXPERT_2.0 (C2799465497-POCLOUD) podaac-data-subscriber - Partial PASS - (see job status https://harmony.earthdata.nasa.gov/jobs/f6359e35-2a9d-429d-a0ae-dc6ff2571e1b) - 12 of 14 granules passed L2SS-py steps as expected podaac-data-downloader - Partial PASS (see job status: See https://harmony.earthdata.nasa.gov/jobs/2f22af24-76fb-491e-93d3-f179155bb8ff) - 12 of 14 granules passed L2SS-py steps Harmony - request - Partial PASS - some L2SS-py steps does not succeed in Harmony - 1 of 3 granules passed L2SS-py https://harmony.earthdata.nasa.gov/C2799465497-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2024-03-24T18:00:00%2B00:00":"2024-04-01T07:00:00%2B00:00")&skipPreview=true&concatenate=true (see JobStatus: https://harmony.earthdata.nasa.gov/jobs/5fe55c56-0194-46ff-957e-3d9d6c60f6b2)

Update: (5/20/24) - subscriber/downloader not initiating Harmony Job as expected in UI dashboard

Harmony UAT- v 0.0.1029 (l2ss-py:2.10.0rc8): (C1261072646-POCLOUD) - FAILED Harmony UAT - API request - FAIL - 1 granule Request: https://harmony.uat.earthdata.nasa.gov/C1261072646-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-05-14T18:00:00%2B00:00":"2023-11-18T07:00:00%2B00:00")&skipPreview=true See Job status: https://harmony.uat.earthdata.nasa.gov/jobs/ea333348-877d-42c7-8b77-47f347373a9d

Harmony UAT JOB: Job in Harmony-uat-1 OPS Jobs: Jobs in Harmony-ops-1

Harmony-ops-job-completewitherrors-1

davidcolemanjpl commented 1 month ago

Re-test in Harmony UAT- v0.0.1035 l2ss-py:2.10.0rc8

Used '9' SWOT_L2_LR_SSH_Expert (C1261072646-POCLOUD) granules for subsetting: Test FAILED

Harmony API Request: Failed https://harmony.uat.earthdata.nasa.gov/C1261072646-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-01-01T18:00:00%2B00:00":"2024-12-21T07:00:00%2B00:00")&skipPreview=true

See Harmony JobStatus: https://harmony.uat.earthdata.nasa.gov/jobs/06187ad2-1c06-43e1-bea2-6283e58cb830

Respective Curl request: Failed % curl --location --request GET 'https://harmony.uat.earthdata.nasa.gov/C1261072646-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-01-01T18:00:00%2B00:00":"2024-12-21T07:00:00%2B00:00")&skipPreview=true' --header "Authorization: Bearer [Token]"

See Harmony JobStatus: https://harmony.uat.earthdata.nasa.gov/jobs/01446082-a5dc-44a2-bd73-df50da13fab9

Image

davidcolemanjpl commented 1 month ago

Re-test in Harmony (OPS) - v 0.0.1036 Data subscriber - v1.15.2 L2SS-Py - v2.10.0

SWOT_L2_LR_SSH_EXPERT_2.0 (C2799465497-POCLOUD) podaac-data-subscriber - Partial PASS - (see job status https://harmony.earthdata.nasa.gov/jobs/775bac04-a718-4b7d-8aaa-b436245ce4b2) - 12 of 14 granules passed L2SS-py steps

podaac-data-downloader - Partial PASS (see job status https://harmony.earthdata.nasa.gov/jobs/7ff8a8e0-d234-4197-8b7e-c252d4e60fa0) - 12 of 14 granules passed L2SS-py steps

Harmony - request - Partial PASS - some L2SS-py steps do not succeed in Harmony - 1 of 3 granules passed L2SS-py API Request: https://harmony.earthdata.nasa.gov/C2799465497-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2024-03-24T18:00:00%2B00:00":"2024-04-01T07:00:00%2B00:00")&skipPreview=true&concatenate=true

(see JobStatus:https://harmony.earthdata.nasa.gov/jobs/168a334a-5acf-4e8a-a231-7ae5494d56f6)

OPS-L2SS-py-errors-1

davidcolemanjpl commented 1 month ago

Re-test in Harmony UAT- v0.0.1043 l2ss-py:2.10.0rc8

SWOT_L2_LR_SSH_Expert (C1261072646-POCLOUD): Harmony API Request: '9' of '9' granules Failed subsetting: https://harmony.uat.earthdata.nasa.gov/C1261072646-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2023-01-01T18:00:00%2B00:00":"2024-12-21T07:00:00%2B00:00")&skipPreview=true

See recent Harmony JobStatus: https://harmony.uat.earthdata.nasa.gov/jobs/cdaf9426-8225-46f4-bdcc-4d8c00ac6c20

___

Re-test in Harmony (OPS) - v 0.0.1036 Data subscriber - v1.15.2 L2SS-Py - v2.10.0 / concise:0.8.0

SWOT_L2_LR_SSH_EXPERT_2.0 (C2799465497-POCLOUD) podaac-data-subscriber - PASS - (see job status https://harmony.earthdata.nasa.gov/jobs/89cce66d-b83f-404d-8067-827a35480170) - 7 of 7 granules passed L2SS-py steps

podaac-data-downloader - PASS (see job status https://harmony.earthdata.nasa.gov/jobs/adc60907-8d1f-40ff-a47d-1bfb017f56bf) - 7 of 7 granules passed

Harmony - API request - Partial PASS - some L2SS-py steps do not succeed in Harmony - (1 of 3 granules passed L2SS-py) Request: https://harmony.earthdata.nasa.gov/C2799465497-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?forceAsync=true&subset=lat(34.84931:35.83342)&subset=lon(-125.57812:-124.73438)&subset=time("2024-03-24T18:00:00%2B00:00":"2024-04-01T07:00:00%2B00:00")&skipPreview=true&concatenate=true

jobStatus: https://harmony.earthdata.nasa.gov/jobs/b9bd1a29-9b75-4356-9e34-0be37a01d0b4