search

snowex-hackweek / jupyterhub

jupyterhub configuration for snowex hackweek 2021
https://snowex.hackweek.io
MIT License
1 stars 0 forks source link

Server fails to start if home directory > PVC quota (10GB) #7

Open scottyhq opened 3 years ago

scottyhq commented 3 years ago

We're using a default capped home directory volume size of 10GB. If a user fills up this space and shuts down their server, it fails to start when they next try to log in, with logs showing:

  Normal   Pulled                  6s                kubelet                  Successfully pulled image "uwhackweek/snowex:latest" in 1.007546596s
  Warning  BackOff                 4s (x5 over 48s)  kubelet                  Back-off restarting failed container
➜  new kubectl logs -n jhub          jupyter-jacktarricone
/srv/conda/envs/notebook/lib/python3.8/subprocess.py:848: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
[I 2021-06-03 23:58:18.598 LabApp] JupyterLab extension loaded from /srv/conda/envs/notebook/lib/python3.8/site-packages/jupyterlab
[I 2021-06-03 23:58:18.598 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab
Traceback (most recent call last):
  File "/usr/local/bin/repo2docker-entrypoint", line 97, in <module>
    main()
  File "/usr/local/bin/repo2docker-entrypoint", line 78, in main
    tee(chunk)
  File "/usr/local/bin/repo2docker-entrypoint", line 64, in tee
    f.flush()
OSError: [Errno 28] No space left on device

Docs on storage optimizations are here https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-storage.html?highlight=volume%2010GB#customizing-user-storage

Some solutions:

  1. Increase the home directory volume size for all users https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-storage.html?highlight=volume%2010GB#size-of-storage-provisioned.

  2. manually resize the user PVC in aws console (not sure if k8s will complain about state mismatches for this)

  3. kubectl edit pvc $your_pvc and modify spec.resources.requests.storage https://stackoverflow.com/questions/40335179/can-a-persistent-volume-be-resized

scottyhq commented 3 years ago

tried 3. but ran into: error: persistentvolumeclaims "claim-jacktarricone" could not be patched: persistentvolumeclaims "claim-jacktarricone" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

kubectl describe storageclass

Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

Needed to first run kubectl edit sc and add allowVolumeExpansion: true