How to start a Jupyter server on a compute cluster?

[jan-janssen]

There are two ways:

Run Jupyter notebook server on the login node

Step: 0 - Login to your cluster

You should be able to already login to your cluster using ssh:

ssh cluster 

Step: 1 - Start the Jupyter notebook

Navigate to your calculation directory and start a jupyter notebook:

cd ~/my/projects
jupyter notebook --no-browser

It is important to keep this terminal open as once it is closed the jupyter notebook is shutdown.

It should return something like:

Or copy and paste one of these URLs:
        http://localhost:8888/?token=bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd
     or http://127.0.0.1:8888/?token=bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd

The important part here is the <port> the jupyter notebook is running on - in this case 8888 and the token to access the jupyter notebook - in this case bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd.

Step: 2 - Establish the connection to the Jupyter notebook

Then you can open a port on your local computer to connect to this jupyer notebook session. For this a second terminal is opened on your local computer:

ssh cluster -L 9000:localhost:<port> 

The <port> is the port returned by the jupyter notebook command in the previous step.

Step: 3 - Connect to the Jupyter notebook

Finally, you can access the jupyter notebook on your local web browser using https://localhost:9000 . It is going to request the token which was returned in step 1.

Run Jupyter notebook server on one of the compute nodes

Step: 0 - Login to your cluster

You should be able to already login to your cluster using ssh:

ssh cluster 

Step: 1 - Configure the connection from the login node to the compute node

For the scripts typically used a bin folder is created in the home directory:

mkdir ~/bin

Create the script in ~/bin/login_to_interactive.sh:

#!/bin/bash
ssh -L <port>:localhost:8888 $(squeue -u $USER | grep interact | awk '{ print $8 }')

Replace <port> with a unique port, which is unlikey to be used by others, commonly a five digit number.

Set the executable bit:

chmod +x  ~/bin/login_to_interactive.sh

Step: 2 - Request an interactive allocation

These commands might differ between different HPC clusters. Submit a request for an interactive session:

salloc -N1 -t 120 --reservation=debug --qos=debug --partition=debug

Step: 3 - Start the Jupyter notebook

Inside this session navigate to your calculation directory and start a jupyter notebook:

cd ~/my/projects
jupyter notebook --no-browser

It should return something like:

Or copy and paste one of these URLs:
        http://localhost:8888/?token=bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd
     or http://127.0.0.1:8888/?token=bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd

The important part here is the token to access the jupyter notebook - in this case bf0cdb1a05142b261760cb0f030656ec95e97beb8c6e34bd.

Step: 4 - Establish the connection to the Jupyter notebook

Then you can open a port on your local computer to connect to this jupyer notebook session. For this a second terminal is opened on your local computer:

ssh cluster -L 9000:localhost:<port> -t '~/bin/login_to_interactive.sh'

The <port> is the same port defined in the ~/bin/login_to_interactive.sh script.

Step: 5 - Establish the connection to the Jupyter notebook

Finally, you can access the jupyter notebook on your local web browser using https://localhost:9000 . It is going to request the token which was returned in step 3.

[jan-janssen]

@aageo25 and @pkruzikova - I hope this helps.