Create xpublish routers for building APIs around xarray datasets

[mpiannucci]

Currently, there are suites of services included in THREDDS that are relied upon by users to access data. This issue tracks some attempts to replicate the experiences users expect when requesting and viewing remote xarray datasets from outside of python notebooks.

To accomplish this, we will be building xpublish routers that can be modularly composed together to create the service suite a user would need or want to expose their data to other users. This effort started at the IOOS CodeSprint 2022 (Original prototypes from that event here: https://github.com/asascience/restful-grids) Below is a services wish list of sorts as we compile and build out prototypes:

• [x] OpenDAP (via https://github.com/gulfofmaine/xpublish-opendap)
• [x] EDR (via https://github.com/gulfofmaine/xpublish-edr)
• [x] WMS (via https://github.com/asascience-open/xpublish-wms)
• [ ] Catalog Services (a la browsing datasets on Thredds or other catalog html site)

Other service suggestions are welcome!

[mpiannucci]

cc @abkfenris

[abkfenris]

I haven't gotten to play with your router yet, but I did just find rio-tiler which sits under titiler, which may allow you to factor out some of the tiling functionality that they have already implemented.

[mpiannucci]

I haven't gotten to play with your router yet, but I did just find rio-tiler which sits under titiler, which may allow you to factor out some of the tiling functionality that they have already implemented.

Yeah i found that too.. They actually tile in the exact same way as I was, which is a little unfortunate because I was hoping that they would have a more efficient method.

[mpiannucci]

I am also building a prototype frontend for these routers to be used via the browser here: https://github.com/asascience-open/xreds

It is very much a work in progress but will build it in the open

[abkfenris]

Nice! This is the kind of thing I'm thinking about for Xpublish distros, which speaking of I've got a draft of a more public post in the hackmd doc to try to get others interested.

Some so we don't have to keep re-inventing the wheel

https://github.com/NCEI-NOAAGov/zarrdap/blob/24a7c5a7fd0fec055cacff21b6fa977cb5b1de02/app/api_dap2/backend.py#L81-L93

👀

https://github.com/asascience-open/xreds/blob/0aa2474e7def7f96791c1443a9547cc39b907823/xreds/server.py#L30-L48

[abkfenris]

I haven't run it yet to see what you've got, but an idea for the frontend to use hvplot explorer to be able to take a quick look at datasets, at least until building out more integrated exploration tools (panel in FastAPI).

[mpiannucci]

So far, it's basically a simplistic Godiva clone: it uses the wms endpoint to display layers on a map and you can click to query data at given points. You can choose your timestep, color ranges, and color palette. Right now memory struggles a bit because it's not setup with dask or anything.

My plan is for this to be the hub for kind of a whole bunch of systems running on our liver eyes cluster, allowing for control of ingest systems, QAQC, JUPYTERHUB, opendap frontend, wms frontend. I started with wms frontend because it's the easiest of all of these to build in a silo while also showing that we can indeed use the data straight from NODD and kerchunk without thredds or any other aggregation services.

I am not very knowledgeable about hvplot or any frontend python tools, I pretty much always build the data devices in python but the frontend services in JavaScript.

Excited to keep this coversatikn moving along and happy to collaborate.

[aufdenkampe]

@mpiannucci, to meet your "Catalog Services" need, I wonder if you could lift some code from or integrate with ZarrDAP, which implements a custom HTML ZarrDAP Catalog and from it generates an Intake catalog.

[mpiannucci]

Yeah that sounds like a good approach. We will be meeting with the zarrdap folks this week and talking about possible collaboration and integrations

[mpiannucci]

Can we build routers for generating STAC or Intake catalogs on the fly?