search

asascience-open / restful-grids

Exploring modern RESTful services for gridded data
https://asascience.github.io/restful-grids/
MIT License
10 stars 3 forks source link

What's next? #19

Open abkfenris opened 2 years ago

abkfenris commented 2 years ago

So now that the code sprint is over, what's next for this project?

Personally, I think we've made an awesome start, and shown just what is possible with xpublish, even if we didn't quite manage ~the holy grail~ to generate Zarr datatree pyramids on the fly.

I'm going to suggest that we spend some time getting the other routers cleaned up and build some documentation about each of them, so that we can have something to share with the IOOS/Pangeo/xarray communities.

We can also document where we are at with the datatree router, what are approaches and findings so far are for sharing too.

In the end, I see this repo potentially becoming a testing ground for new routers, and once they are proved possible, we can migrate them to their own repos, so that we end up with xpublish-edr, xpublish-xyz, xpublish-wms, xpublish-pyramid and so forth.

mpiannucci commented 2 years ago

I love the idea of treating this repo as a playground for developing news types of routers until they are stable. Then creating true independent packages when/if they become fully baked.

Even if they never become production ready, having a whole bunch of examples of how to build common geospatial rest services on top of xarray is super valuable

mpiannucci commented 2 years ago

For me personally, I will be continuing to play with developing web mapping services (tiles, wms, zarr tile pyramid) to see how far towards a production level of speed I can get.

abkfenris commented 2 years ago

Is anyone interested in showing what we've done off to the Pangeo community meeting tomorrow? https://pangeo.io/meeting-notes.html

jonmjoyce commented 2 years ago

I'm planning to attend the Pangeo meeting tomorrow. How do we get on their agenda?

abkfenris commented 2 years ago

Probably the best way is to post in their thread that you've got more to show. Not sure if they have any agenda topics queued up yet: https://discourse.pangeo.io/t/pangeo-weekly-telecon/134/140

jonmjoyce commented 2 years ago

I had the times wrong for the Pangeo meeting, so looks like I have a conflict tomorrow. I'll try to put something together for a future Pangeo meeting. Of course, if you want to mention the project, feel free! Thanks.

abkfenris commented 2 years ago

I reported back a little about what we've done recently to the Pangeo meeting, though it seems like there may have been enough progress on datatree, tiles, and WMS that I didn't have a great handle on it.

I started a package for an EDR router. https://github.com/gulfofmaine/xpublish-edr/

I'm playing with something that might be a bit rediculous, but I'm using entry_points to allow extending the formats that the EDR API endpoints can respond in via additional packages. So there could be something like xpublish-edr-hvplot that could generate plots, or xpublish-edr-grib if I want to question my sanity further.

abkfenris commented 2 years ago

I spun out https://github.com/gulfofmaine/xpublish-opendap today and I think I found the issue that was making much more inefficient then necessary (bad query constraint parsing).

abkfenris commented 2 years ago

Anyone want to chat about the tile and other work at the Pangea meeting tomorrow? I've demoed the EDR and OpenDAP routers but I don't know what's going on in the guts of tile generation as well.

I'm planning on joining the meeting, but I'm mainly gonna be focused on another topic.

mpiannucci commented 2 years ago

I have never joined one of the meetings, but I will try to join (I am really new to this open source geo world)... On the fly zarr tiling proof of concept is working really well. Its definitely not perfectly efficient but it works well enough to demo and for a starting point to make it more efficient down the road.I am already working to see how far I can push it with some internal datasets with good results. Extending xpublish is really powerful.

abkfenris commented 2 years ago

Very cool to hear the routers are working well. I'm nearly ready to punt a lot of the usual array of IOOS services into the sun for a project that I'm currently working on, so I'm eyeing that WMS or tree router.

They start the Pangeo meetings with '60 second updates' that anyone/everyone can mention some things they are working on and if they are new get introduced to the community.

Then there is ostensibly a planned agenda time after the updates. If you want to demo those routers and really have a discussion that would be the time to do it. You can volunteer for an agenda slot in the Discourse topic I linked above, or sometimes if folks are interested in someone's 60 second update it naturally moves into the agenda and discussion. They are usually pretty low relaxed meetings even though they are some wicked smart folks doing a ton of really cool work.

abkfenris commented 1 year ago

@ocefpaf suggested that maybe we should demo this and what it's evolved into for a DMAC webinar. Anyone interested in in that?

mpiannucci commented 1 year ago

I could probably help out with that, I have been building on this a bit for some internal projects (in another repo for the time being) and have started using it with kerchunked datasets recently

abkfenris commented 1 year ago

Oh it would be cool to hear about those too. I'm also got some that I've been playing with for the next gen of NERACOOS data infrastructure.

Maybe we should find a chance to chat first?

mpiannucci commented 1 year ago

Yeah Id love to chat about it. Maybe sometime later this week (thursday?) if youve got the time

abkfenris commented 1 year ago

Thursday I'm pretty flexible till late afternoon Eastern.

mpiannucci commented 1 year ago

Cool ill send a meeting invite soon