increase persisted user space ?

[rsignell-usgs]

Okay, I checked out the shiny new jupyterhub at https://esiphub2.ndslabs.org!

I was able to login with github credentials and switch between jupyterlab and regular jupyter environments https://esiphub2.ndslabs.org/user/rsignell-usgs/lab https://esiphub2.ndslabs.org/user/rsignell-usgs/tree

So then I modified my ~/.condarc to:

channels:
  - conda-forge
  - defaults
envs_dirs:
  - /home/jovyan/my-conda-envs/

and restarted the server to make sure my ~/.condarc was persisted. It was! :champagne:

I then tried to create a custom datashader environment:

$ wget https://raw.githubusercontent.com/bokeh/datashader/master/examples/environment.yml

and ran out of room!

...
paramnb-2.0.2- 100% |###########################################| Time: 0:00:00  10.17 MB/s
holoviews-1.9. 100% |###########################################| Time: 0:00:03   1.03 MB/s
geoviews-1.4.3 100% |###########################################| Time: 0:00:00   2.62 MB/s
ERROR conda.core.link:_execute_actions(337): An error occurred while installing package 'conda-forge::xerces-c-3.2.0-0'.
OSError(28, 'No space left on device')
Attempting to roll back.

So it looks like we have only 1GB on /home/jovyan, is that right?

jovyan@jupyter-rsignell-2dusgs:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          39G   11G   28G  29% /
tmpfs           2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/rbd1       976M   19M  891M   3% /home/jovyan
/dev/vda1        39G   11G   28G  29% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/firmware

We will need some solution to persisted storage here since most of our custom environments are 1GB or larger. I would say we need at least 30GB per user.

[craig-willis]

@rsignell-usgs Sorry, I didn't realize until the call today that we were using issues on this repo. I'll let you know when this config change is implemented.

[rsignell-usgs]

@craig-willis , fantastic. BTW, Congrats on the 👶 !!

[craig-willis]

I've updated the singleuser config as follows and restarted the chart.

singleuser:
  memory:
    guarantee: 1G
    limit: 2G
  storage:
    type: dynamic
    capacity: 5Gi
    dynamic:
      storageClass: rook-block
  image:
    name: jupyter/scipy-notebook
    tag: latest

Note that the test cluster currently has limited storage -- I've only provisioned ~40GB storage total for testing. If we need to increase this in the short term, let me know.

[rsignell-usgs]

@craig-willis, I just looked, and was expecting to see increased space in /home/jovyan, but I'm not seeing it:

$jovyan@jupyter-rsignell-2dusgs:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          39G   19G   20G  49% /
tmpfs           2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        39G   19G   20G  49% /etc/hosts
/dev/rbd0       976M  210M  700M  24% /home/jovyan
shm              64M     0   64M   0% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/firmware

Still looks to be 1GB. I tried making my datashader environment and it failed again, running out of room.

Perhaps since this is primarily to be used for training, perhaps we don't need too much local room?

But it would be nice to have at least 5GB to test out a custom env or two...

[craig-willis]

@rsignell-usgs My mistake, when changing this setting under JupyterHub on Kubernetes I have to delete any existing persistent volume claims (which will delete your user data). I've reduced the value to 5G per user for now (we can certainly make this bigger if we resize this cluster later).

For this to take effect, I need to delete the PVC associated with your account. Let me know if this is OK. The next time you start a notebook, you'll get the 5GB disk.

[rsignell-usgs]

That’s totally fine. Blow it away!

[craig-willis]

Deleted. Next time you start the server you should get a 5GB volume. Let me know if otherwise.

[rsignell-usgs]

@craig-willis, I hope you are messing around, or maybe I killed it.

I create a custom env and ran this notebook, and seemed to be running fine, but then got error. And now https://esiphub.ndslabs.org/hub/home is returning 504.

Was looking good there for a while....

[craig-willis]

@rsignell-usgs Not sure what happened -- I wasn't doing anything with the system yesterday. The 504 from Jupyterhub usually signals that the server is under load. These instances are small (2 core 4GB ram) -- so maybe we're to the point that we need to rescale the cluster for more realistic use?

[rsignell-usgs]

Indeed, 4GB ram could be problematic for the datashader/trimesh grid stuff. I know that Unidata is running their JupyterHub on a XLarge instance on Jetstream, which is 60GB RAM.

[craig-willis]

Yeah, it sounds like we're outgrowing the initial setup. It would be helpful to start collecting some of these requirements both for the core ESIPhub team (I'm assuming 10 users for now) and estimates for the workshop instance -- I've started in https://github.com/nds-org/esiphub/issues/6.

With the Kubernetes deployment, we'll have both resources at the node level and limits put on each user's running container. Do you have a sense of what would be required for a single user in this case for ESIPhub? We can certainly scale up the nodes to something larger, but will likely still want to know what limits to put on individual users.

[rsignell-usgs]

On pangeo.pydata.org, the controller node has 4GB RAM, and the worker nodes have 6GB RAM.

Closing this since the user storage is now 5GB.