search

coiled / examples

Examples using Dask and Coiled
18 stars 3 forks source link

Pangeo example #4

Open mrocklin opened 1 year ago

mrocklin commented 1 year ago

@ncclementi any thoughts on a good Pangeo-y example we have lying around?

ncclementi commented 1 year ago

The one on the main page of our geospatial use cases looks pretty and simple: https://coiled-new-clean.webflow.io/use-cases/geospatial

In that page we also have linked

mrocklin commented 1 year ago

@jrbourbeau I'm curious, what are the odds you think you'll find something solid by end of week?

jrbourbeau commented 1 year ago

There's the issue over in coiled/benchmarks where I'm soliciting representative Pangeo workloads. At the moment I'm waiting for a specific suggestion from the Pangeo folks involved in that issue.

I also just asked if https://gallery.pangeo.io/repos/pangeo-gallery/physical-oceanography/01_sea-surface-height.html or https://gallery.pangeo.io/repos/pangeo-gallery/cmip6/global_mean_surface_temp.html seemed representative to them. I think porting one of those examples over to Coiled would be straightforward.

mrocklin commented 1 year ago

Yeah, sea surface altitude is my usual go-to. A challenge though is that that data is on GCP. Maybe it's easy to lift and shift that data over to AWS though.

We could also use multiple accounts to get multi-cloud. I'd like to avoid the complication when doing demos though. Who knows though, maybe that's a good selling point.

jrbourbeau commented 1 year ago

A challenge though is that that data is on GCP. Maybe it's easy to lift and shift that data over to AWS though.

I dug a little bit a found that pangeo folks have already lifted and shifted for the cmip6 dataset:

import intake
col = intake.open_esm_datastore("https://cmip6-pds.s3-us-west-2.amazonaws.com/pangeo-cmip6.json")

We could also use multiple accounts to get multi-cloud

I was chatting with Nat about this offline earlier today. This is doable, but maybe not actually needed at the moment given the above comment ^

mrocklin commented 1 year ago

Ooh, nice find!

On Wed, Apr 12, 2023, 5:02 PM James Bourbeau @.***> wrote:

A challenge though is that that data is on GCP. Maybe it's easy to lift and shift that data over to AWS though.

I dug a little bit a found that pangeo folks have already lifted and shifted for the cmip6 dataset:

import intakecol = intake.open_esm_datastore("https://cmip6-pds.s3-us-west-2.amazonaws.com/pangeo-cmip6.json")

We could also use multiple accounts to get multi-cloud

I was chatting with Nat about this offline earlier today. This is doable, but maybe not actually needed at the moment given the above comment ^

— Reply to this email directly, view it on GitHub https://github.com/coiled/examples/issues/4#issuecomment-1506017967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTFZFATMXXRE7AFWX4DXA4RAHANCNFSM6AAAAAAWWWDNRY . You are receiving this because you authored the thread.Message ID: @.***>

jrbourbeau commented 1 year ago

Given cmip6 is on AWS, I think it should be straightforward to port over https://gallery.pangeo.io/repos/pangeo-gallery/cmip6/global_mean_surface_temp.html. I'll have some time to focus on that tomorrow unless you beat me to it @mrocklin

mrocklin commented 1 year ago

Playing with this now @jrbourbeau unless you're already on it

mrocklin commented 1 year ago

I've got global surface mean temperature, but it's not a very good demo. It's a lot of fussing about with a single nice graph at the end. I may look a little deeper, but I'm not super excited yet.