NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a developer, I want to investigate, understand, and implement a reader for the WRDS HEFS service #138

Open epag opened 2 months ago

epag commented 2 months ago

Author Name: Hank (Hank) Original Redmine Issue: 121930, https://vlab.noaa.gov/redmine/issues/121930 Original Date: 2023-10-26


The URL is:

https://nwcal-wrds.[domain]/api/hefs/v1/

Swagger:

https://nwcal-wrds.[domain]/api/hefs/swagger/

I would like to investigate the API and gain an understanding relatively soon. We can then decide if and when we want to write a reader for it. Marking this as high priority, but not urgent,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-10-26T15:58:33Z


Fixed a typo.

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-02T16:56:56Z


We are waiting on this until a user asks for it, but doing an initial investigation sooner makes sense.

This might end up becoming a framework picked up for the other services down the road.

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-02T17:37:41Z


Here is an example URL:

https://nwcal-wrds.[domain]/api/hefs/v1/series/?limit=10&start_date=2023-10-01

Here is the example output:

{
    "count": 6895131,
    "next": "http://nwcal-wrds.[domain]/api/hefs/v1/series/?limit=10&offset=10&start_date=2023-10-01",
    "previous": null,
    "results": [
        {
            "id": "b939ffe0-a3e4-4dd6-9c0d-d2f325dbf874",
            "created_at": "2023-10-11T18:57:48.449916Z",
            "updated_at": "2023-10-11T18:57:48.449940Z",
            "deleted_at": null,
            "is_deleted": false,
            "type": "instantaneous",
            "location_id": "PDLC2",
            "parameter_id": "QINE",
            "ensemble_id": "MEFP",
            "ensemble_member_index": 1951,
            "time_step_unit": "second",
            "time_step_multiplier": 21600,
            "start_date_date": "2022-05-16",
            "start_date_time": "00:00:00",
            "end_date_date": "2022-06-15",
            "end_date_time": "00:00:00",
            "forecast_date_date": "2022-05-16",
            "forecast_date_time": "00:00:00",
            "miss_val": "nan",
            "station_name": "PDLC2R",
            "lat": 38.4872222222,
            "lon": -105.373055556,
            "x": null,
            "y": null,
            "z": null,
            "units": "CFS",
            "creation_date": "2022-05-16",
            "creation_time": null
        },
        {
            "id": "dc822be7-a136-4845-b472-00f65d5cf7fa",
            "created_at": "2023-10-11T18:57:48.498368Z",
            "updated_at": "2023-10-11T18:57:48.498377Z",
            "deleted_at": null,
            "is_deleted": false,
            "type": "instantaneous",
            "location_id": "PDLC2",
            "parameter_id": "QINE",
            "ensemble_id": "MEFP",
            "ensemble_member_index": 1952,
            "time_step_unit": "second",
            "time_step_multiplier": 21600,
            "start_date_date": "2022-05-16",
            "start_date_time": "00:00:00",
            "end_date_date": "2022-06-15",
            "end_date_time": "00:00:00",
            "forecast_date_date": "2022-05-16",
            "forecast_date_time": "00:00:00",
            "miss_val": "nan",
            "station_name": "PDLC2R",
            "lat": 38.4872222222,
            "lon": -105.373055556,
            "x": null,
            "y": null,
            "z": null,
            "units": "CFS",
            "creation_date": "2022-05-16",
            "creation_time": null
        },
</code>

Its providing only metadata, and my URL limited it to the first 10 headers.

Let me take a look at the Swagger,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-02T17:47:21Z


Not a big fan of the Swagger documentation. It may not be fully baked. I've asked David Moring to provide an example URL for a specific scenario that I can use as a starting point for understanding the service.

Thanks,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-02T17:47:39Z


Waiting for a reply,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-02T18:51:28Z


Was told in the WRES/WRDS meeting that they will work that request into examples dovetailed into a Jupyter Notebook they are hoping to release next Wednesday. I also informed them about the lack of a time zone in the response, so that we can't be sure in what time zone dates are presented.

Last point... They intend to move all services to something like the HEFS approach, with the next one likely being the Location Service, which will be moved to the Cloud.

Leaving on hold,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-11-16T21:01:53Z


Here is the Jupyter Notebook (enter the token "WRDSJupyter"):

https://nwcal-wrds.[host]/jupyter/lab/tree/hefs

It should help to investigate how to use the HEFS service. We would then need to write a brand new reader to prepare proper requests and parse the results.

Hank

HankHerr-NOAA commented 4 weeks ago

Latest from Gautam chatted to me today (edited to clean it up)...

HEFS is up and running in UAT. This is going to be our new landing page in the cloud (not pushed up to the cloud yet but the links work):

https://nwcal-wrds-dev01.[domain omitted]/.

heck out this notebook. it shows how to retrieve an ensemble for a location:

https://colab.research.google.com/github/NOAA-OWP/data-service-notebooks/blob/master/HEFS/2_ensemble_plotting_demo.ipynb

I don't think we need to code against it, yet, but we should at least take a look before the next WRES/WRDS meeting (early November, since I canceled the meet tomorrow) so that we can discuss it if we have questions.

Thanks,

Hank