det-lab / jupyterhub-deploy-kubernetes-jetstream

CDMS JupyterHub deployment on XSEDE Jetstream
0 stars 1 forks source link

JupyterHub for CDMS on Jetstream

Instructions for new users

The deployment is available at https://supercdms.jetstream-cloud.org, all members of the SuperCDMS Gitlab organizations have permission to login using their Gitlab credentials.

Install Jupyter Kernels

All the CDMS Jupyter Kernels can be installed in the user environment by opening a terminal and then running:

> install_cdms_kernels

then reload JupyterLab by reloading the webpage. After this the user should see all the available CDMS kernels. This only needs to be run once. And then run again when new releases of the software stack are made available. Sometimes releases are removed from CVMFS, so it possible that installed kernels stop working, in this case the user can remove them by deleting the right folder from:

~/.local/share/jupyter/kernels/

Use Dask for distributed computing

Dask is a Python package that has an array syntax very similar to numpy but under the hood can distribute the computation across many different "workers". In the SuperCDMS JupyterHub deployment on Jetstream, all users by default have access only to the Jupyter Notebook instance. However, they can leverage Dask to launch more workers inside the Kubernetes cluster and parallelize their computation, having access to way more CPU and RAM.

Access to Dask is available via "Dask Gateway", check their documentation to know more. The deployment itself is explained in this tutorial, but this is not necessary for users.

Before testing it, please notify @zonca on the relevant github issue.

In order to test it:

Information about the deployment