[New Hub] Jack Eddy Symposium

damianavila commented 2 years ago

Hub Description

The request is for the launch or use of a research hub with Dask. (Daniel Marsh, one of the co-organizers, wishes to share tutorial notebooks. He plans to use "intake-esm to access all the CMIP6 climate runs which are hosted in zarr format on aws. We can pull down all the time series data and regress it against the solar forcing used to derive the solar response."

Community Representative(s)

@colliand, can you give us the contact information for the community representatives. Looking at the lead issue, it seems the contacts would be Daniel Marsh and/or Ryan McGranaghan. If that is the case, do you have their contact information? Any GitHub handles?

Important dates

Required start date: Jun 1st
Target start date: ASAP
Any important dates for usage:
- Start date for the event: June 6
- End date for the event: June 10
- The active times for the event (e.g., 9am to 5pm US/Pacific) 8 AM MT - 6 PM MT
- How many people will attend the event? ~100-150
- Do you need to need hub infrastructure to be [pre-initialized before the event] Not a requirement, but a nice-to-have

Hub Authentication Type

Other (may not be possible, please specify in comments)

Hub logo information

URL to Hub Image: https://cpaess.ucar.edu/sites/default/files/meetings/2022/nasa-jack-eddy-symposium-2022-860.jpg
URL for Image Link: {{ URL HERE }}

Hub user image

Repository for user image: { REPO LINK IF IT EXISTS }
User image registry: { REGISTRY IF ONE ALREADY EXISTS }
User image tag and name: { NAME AND TAG IF IT EXISTS }

Extra features you'd like to enable

[x] Specific cloud provider or datacenter: AWS
[x] Dedicated Kubernetes cluster
[x] Scalable Dask Cluster

Other relevant information

I presume GitHub auth would be OK but we need to confirm it. It seems they might need to interact with datasets on AWS, so it might make sense to deploy in that cloud provider... From the lead description, it is not clear to me how this hub will be paid so there is some stuff to figure it out in case we need a new AWS land to deploy into...

Hub URL

jackeddy.2i2c.cloud

Hub Type

daskhub

Tasks to deploy the hub

[x] Engineer who will deploy the hub is assigned
[x] Deploy information filled in above
[x] Initial Hub deployment PR:
[x] Administrators able to log on
[x] Community Representative satisfied with hub environment
[x] Hub now in steady-state

rmcgranaghan commented 2 years ago

Thanks, @yuvipanda - confirming that I have access to the shared-readwrite!

dan800 commented 2 years ago

@dan800 - I updated the configurator with the image that has gcsfs after adding it to the environment, so you should be now good to go on this!

@fperez It seems the switch from the pangeo stack to the heliocloud package has meant some of the standard packages need updating. ImportError: Plotting of arrays of cftime.datetime objects or arrays indexed by cftime.datetime objects requires the optional nc-time-axis (v1.2.0 or later) package.

@rmcgranaghan This might be highlighting a problem we have in that these packages (that perhaps started from similar bases) have diverged making working between the disciplines difficult. It also will make sharing packages across the platforms harder than it could be.

rmcgranaghan commented 2 years ago

@dan800 thanks for investigating. I think you've identified an issue that we can work on to improve HelioCloud to be better equipped to connect Helio and climate communities

It is my understanding the @brianthomas plans to update the HelioCloud image to begin from the latest version of the Pangeo image. I suggest we enumerate the missing packages that are needed for Helio-climate collaboration and share with Brian

dan800 commented 2 years ago

BTW, I did try to install it locally for the session but it essentially hung - I ctrl-c'd after 30 minutes: ''' Welcome to HelioCloud DaskHub (notebook) jovyan@jupyter-dan800:~$ conda install -c conda-forge nc-time-axis Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: \ failed

CondaError: KeyboardInterrupt

(notebook) jovyan@jupyter-dan800:~$ '''

yuvipanda commented 2 years ago

I've also opened https://github.com/2i2c-org/jackeddy-image/pull/2 which would base the jackeddy image off the latest pangeo image rather than the heliocloud image. However, this might mean you might lose some of the functionality that comes from the heliocloud image...

yuvipanda commented 2 years ago

@dan800 also, I suggest trying mamba instead of conda to install stuff on the package (it is a drop-in replacement). I just tried !mamba install -c conda-forge nc-time-axis -y and it immediately worked.

damianavila commented 2 years ago

I think we can close this issue by now since the new hub was deployed on time and the event is already finished. Also marking this one as part of the previous iteration/cycle because it belongs to that one.

Thanks for all your help with this one, @yuvipanda!

fperez commented 2 years ago

Huge, huge thanks and kudos to the team on this one, I think it was a great success.

rmcgranaghan commented 2 years ago

Indeed - it would be hard to overstate how brilliant a job you all did. Genuine thanks.

yuvipanda commented 2 years ago

Glad to help, @fperez @rmcgranaghan @colliand @dan800!

2i2c-org / infrastructure