azavea / kubernetes-deployment

Deployment resources and shared knowledge base for deploying Kubernetes
Apache License 2.0
2 stars 2 forks source link

Upgrade JupyterHub chart version to 2.0.0, add ephemeral scratch space to singleuser environment #29

Closed jpolchlo closed 1 year ago

jpolchlo commented 1 year ago

This update moves to the most recent JupyterHub Helm chart release. This move was needed to allow for the addition of an ephemeral volume to give notebooks a scratch volume. This volume is backed by a mount from EBS that has a lifecycle to match the pod it is connected to. It will not persist after a session expires. It is currently sized at 128GB, but that value can be changed if needed.

From a notebook environment, it is accessible at /scratch:

jovyan@jupyter-jpolchlopek-40azavea-2ecom:~$ ls /
bin   dev  home  lib32  libx32  mnt  proc  run   scratch  sys  usr
boot  etc  lib   lib64  media   opt  root  sbin  srv      tmp  var
jovyan@jupyter-jpolchlopek-40azavea-2ecom:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          20G  9.6G   11G  48% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme1n1    252G   24K  252G   1% /scratch
/dev/nvme0n1p1   20G  9.6G   11G  48% /etc/hosts
shm              64M  4.0K   64M   1% /dev/shm
tmpfs            15G  4.0K   15G   1% /run/secrets/eks.amazonaws.com/serviceaccount
tmpfs           7.7G     0  7.7G   0% /proc/acpi
tmpfs           7.7G     0  7.7G   0% /sys/firmware

(This is from an earlier test when I was using 256GB as the default size.)

Closes #20 Connects azavea/noaa-hydro-data#85

jpolchlo commented 1 year ago

This needs a few days of working with it by various parties to make sure that I've not broken something. Looking for a +1 from @rajadain and @vlulla; if y'all can exercise some notebooks on the NOAA cluster, we can find out if there's broken stuff. I've been banging on this, and things are functioning, but more eyes stress-testing more aspects is good. If nobody complains, I'll merge on 12/22, but I'll be happy to get some +1 reviews on this if you do get to use JH on the cluster before then.

jpolchlo commented 1 year ago

This has been thoroughly exercised and is working well. Pushing the button.