search

coiled / examples

Examples using Dask and Coiled
14 stars 3 forks source link

Add NASA `earthaccess` example notebook #24

Closed jrbourbeau closed 10 months ago

jrbourbeau commented 10 months ago

cc @mrocklin

review-notebook-app[bot] commented 10 months ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

mrocklin commented 10 months ago

I'm playing with this now. Thanks @jrbourbeau

Also, more generally I think that we should store these notebooks with outputs cleared. Do you have other thoughts?

jrbourbeau commented 10 months ago

Forgot to mention that this example requires having a Earthdata login (signup is here https://urs.earthdata.nasa.gov)

Also, more generally I think that we should store these notebooks with outputs cleared. Do you have other thoughts?

Yeah, that's my preference too. See https://github.com/coiled/examples/pull/25 for letting pre-commit automate this

mrocklin commented 10 months ago

How long does ds = xr.open_mfdataset(earthaccess.open(results)) take for you? I know it's commented out, but I wanted to see the duration. I started this a while ago and it's been, a while ...

On Tue, Aug 15, 2023 at 11:00 AM James Bourbeau @.***> wrote:

Forgot to mention that this example requires having a Earthdata login (signup is here https://urs.earthdata.nasa.gov)

Also, more generally I think that we should store these notebooks with outputs cleared. Do you have other thoughts?

Yeah, that's my preference too. See #25 https://github.com/coiled/examples/pull/25 for letting pre-commit automate this

— Reply to this email directly, view it on GitHub https://github.com/coiled/examples/pull/24#issuecomment-1679194440, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTADRCHABHWGLG53ELTXVOMIZANCNFSM6AAAAAA3QHLHCM . You are receiving this because you were mentioned.Message ID: @.***>

--

https://coiled.io

Matthew Rocklin CEO, Dask Maintainer

jrbourbeau commented 10 months ago

I'm not sure. Yesterday, after my Coiled cluster timed out after 20 minutes, I stopped the cell. My takeaway is that something else (e.g. kerchunk, parallel=True in xarray, etc) is needed for processing larger datasets.

mrocklin commented 10 months ago

It seemed fine to me. Sorry for the delay. Merge away.

On Wed, Aug 16, 2023 at 2:59 PM James Bourbeau @.***> wrote:

@.**** commented on this pull request.

@mrocklin https://github.com/mrocklin, just checking in here, anything feedback on this example?

— Reply to this email directly, view it on GitHub https://github.com/coiled/examples/pull/24#pullrequestreview-1581326832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTGAAFTYNM5CBYRTY33XVURAPANCNFSM6AAAAAA3QHLHCM . You are receiving this because you were mentioned.Message ID: @.***>

--

https://coiled.io

Matthew Rocklin CEO, Dask Maintainer

jrbourbeau commented 10 months ago

No worries -- happy to follow-up as needed