[New Hub] NASA Cryo Community -- Cryosphere Community Hub

[colliand]

Hub Description

Tasha Snow and Matt Siegfried of the Colorado School of Mines worked with @fperez and @colliand and members of the cryosphere research community on two NASA proposals over Summer 2022. Both of these proposals were recently funded. These projects will support the use of open science workflows, broaden participation in cryo research, and improve community dynamics. 2i2c will deploy a dask hub for use during a series of events during the Year of Open Science (2023) and beyond.

The main purpose of this issue is to gather the technical details necessary for 2i2c's engineering team to set up the hub.

Future events/milestones/workshops/talks in this project (copied from Google doc running notes)

• Dask aware hub by mid-October - training notebooks and recorded hub training by then?
• October 24-28: Joanna giving 15 min talk to the ICESat-2 Science Team in Austin - invite users onto the hub at this time?
• January 8-12: Tasha giving 20-30 min talk to American Meteorological Society in Denver - no user additions
• March: NASA/NSF FOGSS workshop (in person), 100 Greenland Ice Sheet researchers, two-hour workshop
• April: ICESat-2 Science Team meeting (in person), 100 IS2ST researchers, half-day workshop
• June or Sept: ICESat-2 Hackweek (hybrid), 80 participants, support of 1-week-long workshop
• Summer: SnowEx Hackweek (Hub only)
• September: NASA/NSF WAIS Workshop (hybrid), 120 Antarctic researchers, two-hour workshop
• December 2023: AGU Fall Meeting (in-person), 250 participants, half-day workshop and participation in Year of Open + Science events and presentations
• April 2024-Oct 2025: ICESat-2 Science Team meetings only

Community Representative(s)

• @tsnow03
• @JessicaS11 (potentially?)
• @dfelikson

Others who participated in last week's kickoff event and expressed interest in this hub and the surrounding project inlcude:

• @jdmillstein
• @wsauthoff
• @dfelikson
• @jmunroe

Important dates

• Target start date: 2022-10-07
• Required start date: 2022-10-14
• Any important dates for usage:

Hub Authentication Type

GitHub Authentication (e.g., @mygithubhandle)

Hub logo information

• URL to Hub Image: {{ URL HERE }}
• URL for Image Link: {{ URL HERE }}

Can @tsnow03 or @jessicaS11 please share a URL pointing at an image to be used on the splash page that users land on when they arrive to the hub login page? For example, here is the leap hub's landing page.

Hub user image

• Repository for user image: { REPO LINK IF IT EXISTS }
• User image registry: { REGISTRY IF ONE ALREADY EXISTS }
• User image tag and name: { NAME AND TAG IF IT EXISTS }

What software environment image should be used for the NASA Cryo community hub? In our kickoff, we discussed the idea to launch with a hub image that was used for the SnowEX hackweek. Another possibility would be to use (parts or all of) the JMTE hub used heavily by @fperez and collaborators.

Extra features you'd like to enable

• [ ] scalable dask cluster
• [ ] small, medium, large hardware configurations?

What data centre should be used? The hub should be built on the same data centre where large cryosphere data is stored.

Other relevant information

No response

Hub URL

..2i2c.cloud ### Hub Type _No response_ ### Tasks to deploy the hub - [x] Engineer who will deploy the hub is assigned - [x] Deploy information filled in above - [x] Initial Hub deployment PR: - [x] Administrators able to log on - [x] Community Representative satisfied with hub environment - [x] Hub now in steady-state

[tsnow03]

Thank you everyone for your work on this! Here is our status at the moment: We are still confirming Community Leadership names.

Important dates Target start date: First week of October Required start date: Oct. 14, 2023 Any important dates for usage: Listed in future events above

Hub Authentication Type GitHub Authentication (e.g., @MyGitHubHandle)

Hub logo information URL to Hub Image: {{ URL HERE }} URL for Image Link: {{ URL HERE }} Can @tsnow03 or @JessicaS11 please share a URL pointing at an image to be used on the splash page that users land on when they arrive to the hub login page? Almost complete. What is the difference between the URL to Hub Image and Image Link? Do you need both?

Hub user image Repository for user image: { REPO LINK IF IT EXISTS } User image registry: { REGISTRY IF ONE ALREADY EXISTS } User image tag and name: { NAME AND TAG IF IT EXISTS } What software environment image should be used for the NASA Cryo community hub? We would like to use the SnowEx Hub image with parts of the JMTE hub from @fperez if there are useful pieces not already included in the SnowEx image. We need to combine environment.yml files from the ICESat-2 {https://github.com/ICESAT-2HackWeek/website2022/tree/main/conda}, SnowEx {https://github.com/snowex-hackweek/website2022/tree/main/conda}, and JMTE hubs. Do you need a resolved environment.yml file or will a starting point with additional packages listed be suitable?

Extra features you'd like to enable Scalable dask cluster Small and Medium servers for normal users. Large available for Admins only (and potentially larger later, but it isn't needed yet) Data Center: AWS us-west-2

Hub URL: cryo.compute.2i2c.cloud Other Hub URL: .com Will be decided on with the splash image Hub Type: Dedicated Research Cluster

[sgibson91]

What is the difference between the URL to Hub Image and Image Link? Do you need both?

The "URL to Hub Image" is a URL that points directly at the image for us to reference and pull into the splash page. The "Image Link" is a URL you would like the image displayed on the splash page to hyperlink to, e.g., your project homepage.

We can improve the language in this template to make that distinction clearer :)

[tsnow03]

Great, thanks @sgibson91!

Community Leaders: @tsnow03 @dfelikson and potentially @jessicaS11 and/or one other. I'm still waiting to confirm with both.

URL to Hub Image: {{https://github.com/CryoInTheCloud/CryoCloudWebsite/blob/main/cryocloud.png}} URL to Image Link: {{https://github.com/CryoInTheCloud}}

Other Hub URL: www.CryoInTheCloud.com

Other Relevant Information: Definitely need a Python kernel. Will likely also need an R and Julia kernel, but I will confirm that down the road. It isn't required now. Desktop and Share tools requested

Repository for user image: {https://github.com/CryoInTheCloud/CryoCloudWebsite/tree/main/conda} This will be where we can put the image, but nothing is here at the moment

We are still nailing down the wording for the Funders that will be listed on the splash page

[tsnow03]

Here's a few more odds and ends.

This is our final list of Community Organizers for now: @tsnow03 @JessicaS11 @dfelikson

We would like to list funders as: NASA Transform to Open Science program {https://science.nasa.gov/open-science/transform-to-open-science}, NASA Cryosphere program {https://earth.gsfc.nasa.gov/cryo}, NASA ICESat-2 Science Team {https://icesat-2.gsfc.nasa.gov/science_definition_team}

[colliand]

Thanks @tsnow03 ! I @damianavila to see if he is aware of any other input required from @tsnow03 or other community organizers to set this hub up. I anticipate our engineers will need to share some input with Cryo folks to arrange for the DNS routing to make the hub appear at cryointhecloud.com.

[damianavila]

@colliand @tsnow03, reading this whole thread, it seems we need to deploy a daskhub in AWS land. Do we have an existing AWS account from the Cryo community we can use to deploy this hub? Or this will be an AWS account managed by us? More details about the differences live here: https://infrastructure.2i2c.org/en/latest/topic/cloud-auth.html#aws-access

[colliand]

According to my notes (please correct me if I am wrong @tsnow03!), 2i2c will manage the cloud account with costs passed through to be paid out of the Cryo community grant.

[tsnow03]

Yes that is correct! There may be circumstances down the line where we use someone else's account for credits to be added, but 2i2c will manage our main line.

We have www.cryointhecloud.com in hand. We also have our procurement number finally and are trying to get the next instruction on what they need to do to get a service agreement going. Procurement has been moving very slowly for some reason. We can put our first bill on credit card if they take too long. Let me know when you need money to start and we will get it done by then.

[weiji14]

Hi there, just stumbled upon this while looking at options for an AWS cloud environment. I'm starting an ESIP Machine Learning Tutorial project with @JessicaS11 and @wsauthoff that will be using cloud hosted ICESat-2 data, and would love to contribute to this Cryo Community Hub effort!

image with parts of the JMTE hub from @fperez if there are useful pieces not already included in the SnowEx image. We need to combine environment.yml files from the ICESat-2 {https://github.com/ICESAT-2HackWeek/website2022/tree/main/conda}, SnowEx {https://github.com/snowex-hackweek/website2022/tree/main/conda}, and JMTE hubs.

@tsnow03, is it ok if I start to work on combining the conda environments from ICESat-2 and SnowEx hackweeks at the https://github.com/CryoInTheCloud/CryoCloudWebsite repo? I need an environment with icepyx and a few other Pangeo packages for my project, and this seems like a nice place to put it.

[sgibson91]

Just fyi @tsnow03, regarding the hub image: We provide some documentation and a template repo for automatically building and pushing images, using repo2docker, that are compatible with the infrastructure we will deploy. I realise you already have the https://github.com/CryoInTheCloud/CryoCloudWebsite repo but it may be less effort on your end (or members of the community! Hi Wei! 👋🏻 ) to use this template instead.

[weiji14]

Thanks for the tip @sgibson91! Was going to write a long reply, but decided it might be better to discuss the 'Cryo' repo2docker image in a separate thread (see https://github.com/CryoInTheCloud/CryoCloudWebsite/issues/1). Will let this issue be focused on the JupyterHub 2i2c infrastructure :wink:

[tsnow03]

Thanks @weiji14! That sounds great. I'm happy to help as well.

@sgibson91 It would be great to use this template. I was mimicking the Hackweek GitHub because we want to build a JupyterBook for the group but I have no attachments to anything. I am a nube to this part of the work. We do want all of this in our github so it is all together in one place. I will look through the documents to figure this out.

[sgibson91]

We do want all of this in our github so it is all together in one place.

@tsnow03 No problem! You can place the repo wherever's best for you. We only need the image name and tag to reference from the hub :)

The biggest advantage of our template is that it comes with GitHub Actions workflow files to automatically build and push the image, directly ready for use with the hub when we deploy it.

[tsnow03]

@sgibson91 do you all need our image finished to build everything with AWS or does the building of the docker image happen after that occurs?

[sgibson91]

So long as we have an image to reference, we can deploy the hub, even if it's just a demo one. The image tag can be changed after the fact, either by us directly editing the config, or you will be able to change the image yourself using the configurator: https://docs.2i2c.org/en/latest/admin/howto/configurator.html

[tsnow03]

This is just about ready. Will it be ok for me to make www.hub.cryointhecloud.com our hub link so it will be a subdomain of our main page? I'm not sure what stages you all are at right now and how changing things can cause issues. Also please let me know what I need to do on my end to get things configured properly.

[fperez]

@tsnow03 - I'd suggest making the mnemonic link only hub.cryointhecloud.com (minus the www) so it's a bit shorter. Mentioning it now in case that URL ends up written down somewhere it matters :)

Thx all for the progress!

[sgibson91]

Will it be ok for me to make www.hub.cryointhecloud.com our hub link so it will be a subdomain of our main page?

Given that you control the cryointhecloud.com DNS zone, not us, we will probably create the hub at cryointhecloud.2i2c.cloud and then the steps you will need to take are roughly (depending on your DNS provider) the following:

1. Create a new record under the cryointhecloud.com domain
   • Name the record hub
   • Give the record a CNAME type (as opposed to A record)
2. Point this record at the cryointhecloud.2i2c.cloud URL (or whatever URL we provide you with)

I am asking @damianavila if there is anything else from his side of the process we are waiting on before I can deploy. So we are still in an early phase and can iterate on this.

[tsnow03]

Ok perfect. We will do that. We are almost there on the docker image but need to work out the quay.io side of things to get it there.

[sgibson91]

If you are using the repo2docker-action GitHub Action, then it knows how to push to quay.io and the manual steps for setting that up are documented here: https://docs.2i2c.org/en/latest/admin/howto/environment/hub-user-image-template-guide.html#connect-the-new-repository-to-quay-io

[damianavila]

Small and Medium servers for normal users. Large available for Admins only (and potentially larger later, but it isn't needed yet)

This request probably means having GitHub team memberships and the following setup: https://infrastructure.2i2c.org/en/latest/howto/configure/auth-management/github-orgs.html#restricting-user-profiles-based-on-github-team-membership

@tsnow03, can you confirm this is the configuration you are looking for? Do you have any preexisting GH organization/team?

[colliand]

The GitHub org for this hub is here: https://github.com/orgs/CryoInTheCloud/dashboard

[sgibson91]

The GitHub org for this hub is here: github.com/orgs/CryoInTheCloud/dashboard

Great! Now we just need to know the teams that belong to that org that represent:

• Users allowed to access small and medium servers
• Admins that are additionally allowed to access large servers

[tsnow03]

Great thank you everyone! I'm hung up on the last step to build the docker image to quay.io. I don't think I am getting the repo2docker-action to trigger and push to quay.io with my pull request. Your documentation says NO_PUSH needs to be explicitly disabled. Where does this happen? I'm sure this is easy, but I haven't been able to figure out how to do that.

Great! Now we just need to know the teams that belong to that org that represent:

CryoCloudUser will be the team associated with small medium servers CryoCloudAdvanced will be associated with larger servers

[tsnow03]

I believe we have the docker image in a state you can use. Is there anything else you need from us at the moment for it?

[sgibson91]

Is there anything else you need from us at the moment for it?

• Is it on Docker Hub, quay.io, something else?
• What is the name of the image?
• What tag would you like to use?

Once the hub is up, you can change these at any time using the configurator: https://docs.2i2c.org/en/latest/admin/howto/configurator.html

Thank you!

[tsnow03]

It is on both:

• the image name is: quay.io/cryointhecloud/cryocloudwebsite:2022.10.12
• also in docker: https://hub.docker.com/r/cryointhecloud/cryocloudwebsite/tags
• the tag is 2022.10.12

[sgibson91]

We would like to list funders as: NASA Transform to Open Science program {https://science.nasa.gov/open-science/transform-to-open-science}, NASA Cryosphere program {https://earth.gsfc.nasa.gov/cryo}, NASA ICESat-2 Science Team {https://icesat-2.gsfc.nasa.gov/science_definition_team}

I think right now, our template is configured such that we can only list one funder. I will open an issue to rectify this. Until that is implemented, can we pick just one of these three?

Update: Issue to update our templates to allow multiple funders is here:

• https://github.com/2i2c-org/default-hub-homepage/issues/16

[tsnow03]

Let's start with this one: NASA ICESat-2 Science Team {https://icesat-2.gsfc.nasa.gov/science_definition_team}

[sgibson91]

@tsnow03 Two hubs are now available at the following URLs:

• https://staging.cryointhecloud.2i2c.cloud
  • This is a good place to test out new images before changing them on the prod hub
• https://cryointhecloud.2i2c.cloud
  • This will be the hub you distribute to users

Please check everything is as expected and working. Specifically, please ask someone who is only a member of the CryoCloudUser team to verify which spawn options they see (should be only small and medium), and which spawn options someone who is only a member of the CryoCloudAdvanced team sees (should be small, medium, large and huge).

[sgibson91]

@GeorgianaElena when you've asked communities to setup CNAMES in their own domain, do we have to edit URLs in the hub config and OAuth Apps, or is it ok to leave everything as our 2i2c.cloud domain?

[GeorgianaElena]

@sgibson91, I've used the url provided by the community instead of the 2i2c.cloud one, otherwise https will not work on the cname domain. I believe this is because the cert will be issued with the 2i2c.cloud one, and so, the browser will complain when the URL won't match the info in the ssl cert.

I believe you will need to re-deploy the support chart and/or deleting the pods in the support namespaces when changing the hub domain from the 2i2c.cloud one to the cname. I remember fighting a bit with the support components until things were happy.

[sgibson91]

@GeorgianaElena Ah ok, thank you. Though tbh, we can keep our support stuff in our domain right? E.g. like the pangeo-hubs cluster:

• Support values: https://github.com/2i2c-org/infrastructure/blob/bf6ae45b97d27cf303c68379c35adf9d493097cf/config/clusters/pangeo-hubs/support.values.yaml#L4
• Hub domains: https://github.com/2i2c-org/infrastructure/blob/bf6ae45b97d27cf303c68379c35adf9d493097cf/config/clusters/pangeo-hubs/cluster.yaml#L28

[GeorgianaElena]

Yes I believe

[sgibson91]

@tsnow03 Two hubs are now available at the following URLs:

• staging.cryointhecloud.2i2c.cloud

  • This is a good place to test out new images before changing them on the prod hub

• cryointhecloud.2i2c.cloud

  • This will be the hub you distribute to users

Please check everything is as expected and working. Specifically, please ask someone who is only a member of the CryoCloudUser team to verify which spawn options they see (should be only small and medium), and which spawn options someone who is only a member of the CryoCloudAdvanced team sees (should be small, medium, large and huge).

These are actually not ready. I'm having a spawning issue.

[tsnow03]

Ok. It looks beautiful so far! We are super close. Thank you so much for your work on this!

[sgibson91]

@tsnow03 The hubs are now ready to be tested! We had an auto-scaling issue that has since been resolved. Please make sure to check everything I mentioned in https://github.com/2i2c-org/infrastructure/issues/1702#issuecomment-1283947732

Updated instructions for setting CNAMEs

For the hub to be available at hub.cryointhecloud.com:

1. Please create a CNAME record in your DNS zone
2. Call the CNAME record hub
3. Set the following URL as the CNAME target:
   • cryointhecloud.2i2c.cloud
4. Let me know when you have done that and I shall redeploy the prod hub to use the new URL and update our OAuth apps as well

You can repeat this process to also host the staging hub at staging.hub.cryointhecloud.com, but this time, name the CNAME staging.hub and the target URL is staging.cryointhecloud.2i2c.cloud

Hope that makes sense!

[tsnow03]

Thank you @sgibson91! The hub and staging.hub CNAME records are added now.

I am currently unable to get on the hub, receiving this error for both the hub and staging links:

403: Forbidden Looks like you have NOT been added to the list of allowed users for this hub. Please contact the hub administrators.

[sgibson91]

Hmmm, you are listed as an admin here:

• https://github.com/2i2c-org/infrastructure/pull/1768/files#diff-67a27eea3a5155ba5cdd45181ac7ec6f2870b6f1426ace75b86e60359b082880R47

I can see this in the error logs though:

[W 2022-10-20 16:13:47.315 JupyterHub github:202] User tsnow03 is not in allowed org list

We are restricting access to members of the CryoInTheCloud:CryoCloudUser and CryoInTheCloud:CryoCloudAdvanced teams. Are you a member of those? If not, I don't think you'll be allowed to login, even if you are an admin (though that statement will need to be double checked)

[tsnow03]

Everyone seems to be getting the same error.

[sgibson91]

Are they members of the teams though?

[tsnow03]

Everyone who has tried this out is on the CryoCloudUser team. And I am the one person on both user and advanced teams.

[tsnow03]

[sgibson91]

Can you try following the instructions here please? https://infrastructure.2i2c.org/en/latest/howto/configure/auth-management/github-orgs.html#follow-up-github-organization-administrators-must-grant-access If the OAuth app is not granted the right permissions, it will block everyone

[sgibson91]

@tsnow03 We might have a capitalisation problem in the names of the teams. You provided the team name, which has capitalisation, but we need the team slug, which in this case doesn't have capitalisation, as indicated in the pic below

Is the CryoCloudAdvanced team the same?

[sgibson91]

I am just deploying this change of capitalisation as a shot in the dark

[tsnow03]

Hm yes that is weird, the Teams are capitalized but the handles are not for both of the teams.

For the instructions for granting access, I'm seeing instructions about creating an AuthO app first (we don't have one at the moment). Am I doing the right thing here?

[sgibson91]

For the instructions for granting access, I'm seeing instructions about creating an AuthO app first (we don't have one at the moment). Am I doing the right thing here?

No, I created the OAuth App. You have to grant it the correct permissions in your org settings. But just hang fire and see if this redeploy works

[sgibson91]

@tsnow03 Ok, redeploy has completed, please try logging in again

[sgibson91]

@tsnow03 can you invite me as an admin of the CryoInTheCloud org please? I will check the permissions of the OAuth app