search

2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.
https://infrastructure.2i2c.org
BSD 3-Clause "New" or "Revised" License
105 stars 64 forks source link

[New Hub] OceanHackWeek 2022 #1515

Closed colliand closed 2 years ago

colliand commented 2 years ago

Hub Description

OceanHackWeek 2022 will take place August 15-19. This event is under the umbrella of eScience Institute.

Community Representative(s)

Important dates

Hub Authentication Type

GitHub Authentication (e.g., @mygithubhandle)

Hub logo information

Hub user image

Extra features you'd like to enable

Other relevant information

Hub URL

oceanhackweek.2i2c.cloud

Hub Type

No response

Tasks to deploy the hub

colliand commented 2 years ago

An email was sent to hello@2i2c.org by Alex Kerney today. I've transcribed some of the content into the hub request fields above. Further details need to be processed. I invite 2i2c Engineering Team to consider the questions posed below.

Hi 2i2c,

We’re getting ready for OceanHackWeek 2022, so we thought it might be time to start talking about getting our hub up to speed.

The event is August 15-19 with folks worldwide. We’d like to have at least any major changes settled down by the beginning of August so that instructors can record their tutorials from the same infrastructure as the participants.

It looks like we are shaping up to have around 100 participants or so, and 30ish organizers/instructors/helpers.

Paige (@paigem), Filipe (@ocefpaf), and I (@abkfenris) are on the infrastructure team for OHW, so that probably makes us the community representatives for you.

We’ve got a handful of questions about hub changes/setup:

We’re waiting to hear about some NASA funding. If it comes through before the event, would it be possible to scale up the Dask scaling capabilities, or would that have to be figured out sooner?

Thanks, -Alex Kerney Web Application Developer Gulf of Maine Research Institute

abkfenris commented 2 years ago

Currently our hub is ohw.pilot.2i2c.cloud and it uses Github via AuthO for login. If it's easier to knock things over and stand up a new hub that's ok for us.

We currently have to manually add users via the hub admin, but it would be nice to use Github teams based auth to avoid that chance of mistyping/copying (just had a participant email to ask to get some of their info fixed this morning). @oceanhackweek/participants_2022 and @oceanhackweek/ohw22-organizers (both has sub-teams that should have access) are the teams that we know we would like to give access, but we may have one or two more.

abkfenris commented 2 years ago

I was reminded that our undergraduate program starts a week earlier and goes a week later, so the event is longer (August 8-26), though that is going to be about a tenth of the users as we will have from 15-19 for the HackWeek itself.

damianavila commented 2 years ago

Can we use/switch to Github Teams based auth?

Yes, we can support that.

Our image was huge because of all the different packages required by tutorials and R users that couldn’t be user installed, could we use profiles to have separate images for Python and R for the event?

Yes, that is possible.

How do profiles interact with Dask?

Can you be more specific with this question? The image you are launching should be compatible with Dask Gateway to be able to use it, so it would work on the python image (modulo dask and dask-gateway are installed), but not with the R image, AFAIK. @consideRatio might have more details about this topic.

If we need to update images, does the configurator work with profiles, or would we be making PRs to your infrastructure repo?

The configurator does not work with profiles yet. As a workaround, we may use @sgibson91 bump-jhub-image-action to automatically open the PRs in our infra when you update the images.

Can profiles have different default UIs?

Yes, it should be possible, I think.

How does nbgitpuller work with profiles?

@yuvipanda might have more thoughts about this one. IIRC, there were some issues...

damianavila commented 2 years ago

We’re waiting to hear about some NASA funding. If it comes through before the event, would it be possible to scale up the Dask scaling capabilities, or would that have to be figured out sooner?

If you are planning to use Dask capabilities, this one should be a daskhub from the very beginning.

Currently our hub is ohw.pilot.2i2c.cloud and it uses Github via AuthO for login. If it's easier to knock things over and stand up a new hub that's ok for us.

So, you are OK with decommissioning that hub and creating a new one for the event? You are no longer using that hub?

I presume you are OK about having the hub in a shared cluster (the existing one lives in the 2i2c shared cluster, for instance)?

If that is the case we may tweak the existing hub instead of decommissioning it and creating a new one.

We currently have to manually add users via the hub admin, but it would be nice to use Github teams based auth to avoid that chance of mistyping/copying (just had a participant email to ask to get some of their info fixed this morning). @oceanhackweek/participants_2022 and @oceanhackweek/ohw22-organizers (both has sub-teams that should have access) are the teams that we know we would like to give access, but we may have one or two more.

That should be possible, I think. @sgibson91 can you confirm if our infra is able to manage the subteams mentioned by @abkfenris?

I was reminded that our undergraduate program starts a week earlier and goes a week later, so the event is longer (August 8-26), though that is going to be about a tenth of the users as we will have from 15-19 for the HackWeek itself.

OK, thanks for the update. I have modified the target date to account for that.

consideRatio commented 2 years ago

How does profiles interact with dask?

They are unrelated to what workers you can create via dask gateway. The image used by workers will be the same as the one started via the profile in case different profiles has different images.

The scheduler pods cpu/memory is hardcoded in size/capacity atm so something quite significant to avoid running out of memory.

sgibson91 commented 2 years ago

@sgibson91 can you confirm if our infra is able to manage the subteams

I actually don't know for certain. But I expect membership of subteams is inherited, so all subteams of a team listed in allowed_organizations would be granted access. We could try it and see?

abkfenris commented 2 years ago

Can we use/switch to Github Teams based auth?

Yes, we can support that.

Our image was huge because of all the different packages required by tutorials and R users that couldn’t be user installed, could we use profiles to have separate images for Python and R for the event?

Yes, that is possible.

Fantastic.

How do profiles interact with Dask?

Can you be more specific with this question? The image you are launching should be compatible with Dask Gateway to be able to use it, so it would work on the python image (modulo dask and dask-gateway are installed), but not with the R image, AFAIK. @consideRatio might have more details about this topic.

I wanted to make sure that the dask workers would use the same image as the profile.

If we need to update images, does the configurator work with profiles, or would we be making PRs to your infrastructure repo?

The configurator does not work with profiles yet. As a workaround, we may use @sgibson91 bump-jhub-image-action to automatically open the PRs in our infra when you update the images.

Can profiles have different default UIs?

Yes, it should be possible, I think.

Great. I think our R users would like us more if we send them directly into RStudio. I'm still not sure why....

We’re waiting to hear about some NASA funding. If it comes through before the event, would it be possible to scale up the Dask scaling capabilities, or would that have to be figured out sooner?

If you are planning to use Dask capabilities, this one should be a daskhub from the very beginning.

Ok.

Currently our hub is ohw.pilot.2i2c.cloud and it uses Github via AuthO for login. If it's easier to knock things over and stand up a new hub that's ok for us.

So, you are OK with decommissioning that hub and creating a new one for the event? You are no longer using that hub?

I believe we are ok with decommissioning it. After each event we tell people that they are welcome to keep using it until the funding runs out and we have to shut it down (a different discussion). I don't know if any of last years participants are currently using it, as the 'last activity' time is no longer populating in the admin.

Is there a way to backup everyone directories in bulk? Or should I prompt folks from last year that they should grab anything they may consider precious?

I presume you are OK about having the hub in a shared cluster (the existing one lives in the 2i2c shared cluster, for instance)?

Yes, that works for us to be in the shared infrastructure.

I was reminded that our undergraduate program starts a week earlier and goes a week later, so the event is longer (August 8-26), though that is going to be about a tenth of the users as we will have from 15-19 for the HackWeek itself.

OK, thanks for the update. I have modified the target date to account for that.

Awesome, thanks.

How does profiles interact with dask?

They are unrelated to what workers you can create via dask gateway. The image used by workers will be the same as the one started via the profile in case different profiles has different images.

The scheduler pods cpu/memory is hardcoded in size/capacity atm so something quite significant to avoid running out of memory.

Ok, just wanted to make sure the dask workers would use the same image as the profile.

@sgibson91 can you confirm if our infra is able to manage the subteams

I actually don't know for certain. But I expect membership of subteams is inherited, so all subteams of a team listed in allowed_organizations would be granted access. We could try it and see?

If it does not, I can provide a giant list of teams.

GeorgianaElena commented 2 years ago

Hey @abkfenris! I've started updating the existing ohw hub at https://ohw.pilot.2i2c.cloud as per the new requirements. I've temporarily manually deployed the changes related to authentication and user profiles if you want to give it a try.

I just want to double check a couple of things to make sure I got them right.

Important notes:

abkfenris commented 2 years ago

Hi @GeorgianaElena , Awesome! It looks like I still have access and can use the profiles, and I've asked a few others to test. Right now the Python profile goes to /notebook rather than /lab, but I'm guessing that's because the jupyterhub version is off.

I'm currently working on splitting apart our monster image and getting the dependencies updated, so I'll let you know what those images will be. Depending on how large the images end up, we might go with three profiles (Python, Python + TensorFlow/ML libraries, R).

abkfenris commented 2 years ago

Any chance that we could update our domain? Would oceanhackweek.2i2c.cloud be possible?

I've got a draft image for the python profile if we could git it a shot ghcr.io/oceanhackweek/python:363f75c

abkfenris commented 2 years ago

Can @oceanhackweek/ohw22-mentor-helper-presenter also get access permissions?

abkfenris commented 2 years ago

Ok, I've now got updated images for both Python and R to try:

GeorgianaElena commented 2 years ago

@abkfenris, I've added the new team to the list of allowed teams, changed the profile user images to point to ghcr.io/oceanhackweek/python:e7303ce and ghcr.io/oceanhackweek/r:e7303ce. Also, the hub is now running at https://oceanhackweek.2i2c.cloud 🚀 I believe I have addressed all your comments, but please let me know if I missed anything.

A few questions:

GeorgianaElena commented 2 years ago

Hey @abkfenris. Today, @sgibson91 and me wanted to double-check the sub-teams membership access and this what we discovered:

A workaround for this (apart from having a few very big teams) could be to explicitly allow the sub-teams into the hub.

The currently allowed teams into the hub are:

So, can you please share if any of these bigger teams have sub-teams that should have access? (saying this with my fingers crossed behind my back, hoping they aren't too many 😅 )

I the future, the more sustainable plan is to support this implicit sub-teams membership in the upstream JupyterHub oauthenticator, but I'm afraid it won't be possible for this coming ohw event :(

abkfenris commented 2 years ago

@abkfenris, I've added the new team to the list of allowed teams, changed the profile user images to point to ghcr.io/oceanhackweek/python:e7303ce and ghcr.io/oceanhackweek/r:e7303ce. Also, the hub is now running at https://oceanhackweek.2i2c.cloud 🚀 I believe I have addressed all your comments, but please let me know if I missed anything.

Awesome! Thank you!

  • I noticed that the R profile is throwing a 404 when redirecting to /rstudio correctly. My guess is this is because RStudio isnt' available in the new user image? can you please check and add if it's not already there?

Hmm, it should be in there. I'll do some snooping around this morning.

  • Do you have any intuition about the server resources the users might need during the event? I've set the profiles to be 2 CPU and 8G RAM, but wondering if this will be enough.

It looks like we bumped it to 7/8 GB memory last year. I can't remember which tutorials required that, but then again we don't really have the tutorials nailed down yet...

So, can you please share if any of these bigger teams have sub-teams that should have access? (saying this with my fingers crossed behind my back, hoping they aren't too many 😅 )

Then I won't say it, I'll just give you a list of sub-teams:

abkfenris commented 2 years ago

Ok, lets take a shot at some tweaked images:

abkfenris commented 2 years ago

Wooot! RStudio is working, along with allowing users to create their own conda environments!

There are a few more fiddly bits I want to try to clean up on the images, then then last minute flood of additional packages that our presenters throw our way.

GeorgianaElena commented 2 years ago

Yaaaay 🎉 🎉

@abkfenris, let me know if there's anything else that you need for the hub environment.

Also, if you're planning to use Dask, can you please give that a spin too (if you haven't already), to make sure everything works there too?

There are a few more fiddly bits I want to try to clean up on the images, then then last minute flood of additional packages that our presenters throw our way.

I was hoping to get https://github.com/sgibson91/bump-jhub-image-action to support GitHub docker registry too, so we can have that action open PRs by itself whenever you updated the user images.

When is the event starting? Is it 1August (the "required start date" in the top comment) or August 8-26, 2022?

abkfenris commented 2 years ago

@GeorgianaElena I think it's looking pretty good, but I'm not going to be able to test Dask till next week.

If we have image updates before you get your action updated, is it ok if I make PRs against https://github.com/2i2c-org/infrastructure/blob/ba240a1f424078b2b9d8db55fcd3df87f07dc37d/config/clusters/2i2c/ohw.values.yaml#L45-L59 , or would that mix up part of your workflow?

We're hoping to get our presenters in and testing next week (so hopefully I'm not having to make last minute updates to the images...). The following week we have a small group of participants that will be using it before the main event on the 15-19th.

GeorgianaElena commented 2 years ago

If we have image updates before you get your action updated, is it ok if I make PRs against , or would that mix up part of your workflow?

That would be super nice and very much appreciated @abkfenris, but that's definitely not a requirement. My only concern was the timezone difference and me not being able to update the images in a timely manner. A PR would probably get a lot of more visibility from others on the team 🚀 (or an email to support)

We're hoping to get our presenters in and testing next week (so hopefully I'm not having to make last minute updates to the images...). The following week we have a small group of participants that will be using it before the main event on the 15-19th.

That sounds like a good plan. I believe I will close this issue then, since the hub is a stable state right now and open a new one to track the specific event.

In the meantime, if you notice anything that doesn't work as expected or something is missing, or want to update the user images, please reach out to us through support (https://docs.2i2c.org/en/latest/support.html) to get more visibility and faster response time ✨

GeorgianaElena commented 2 years ago

Just opened https://github.com/2i2c-org/infrastructure/issues/1576 to track the event 🚀