Jupyter notebook server on JADE

[syoffe]

Hello,

Does anyone use a jupyter notebook for analysis directly on JADE, rather than copying data off to a local machine? I was wondering if this is acceptable, before working out if it's possible. I opened a ticket through the service-now portal to ask about jupyter servers on 23rd May, but it was closed unanswered so I thought I'd try here.

Thanks for any advice, Sam

[LiamATOS]

Sam,

The service now portal is just for account issues and software requests.

Hopefully someone here can help as I have seen people running jupyter notebooks on both the compute and login nodes.

Thanks

Liam

[syoffe]

Hi @LiamATOS,

Thanks a lot for your reply. Yeah, I opened the request as a software installation request. I have set up my own jupyter server (with self-signed SSL certificate etc.) on JADE however I do not know the address to connect to. To connect to my server at my institution, I would use https://hostname:port where hostname is the full address I would use to SSH into the machine and port is the port I have selected for the server to run on. If my server is running on node dgj223, what should I use for the hostname (I tried dgj223.hartree.stfc.ac.uk with no success) and what ports would I be allowed to connect to?

Thanks for your help, Sam

[syoffe]

Hello,

Does anyone have any advice on connecting to a jupyter server running on a JADE login node?

Thanks for any suggestions, Sam

[willprice]

I use this bash function to launch a jupyter notebook:

jupyter-server() {
    XDG_RUNTIME_DIR=/tmp/$UID jupyter notebook \
        --port $RANDOM \
        --ip $(hostname) \
        --no-browser
}

I then port forward using SSH tunnels e.g. say I have my jupyter notebook running on dgj223 on port 1234, then I can tunnel from my laptop using:

$ ssh -N -L 8000:dgj223:1234 jade.hartree.stfc.ac.uk

then I can open http://localhost:8000 and the traffic will be forwarded to port 1234 on dgj223 over ssh.

[ghost]

I use this bash function to launch a jupyter notebook:

jupyter-server() {
    XDG_RUNTIME_DIR=/tmp/$UID jupyter notebook \
        --port $RANDOM \
        --ip $(hostname) \
        --no-browser
}

I then port forward using SSH tunnels e.g. say I have my jupyter notebook running on dgj223 on port 1234, then I can tunnel from my laptop using:

$ ssh -N -L 8000:dgj223:1234 jade.hartree.stfc.ac.uk

then I can open http://localhost:8000 and the traffic will be forwarded to port 1234 on dgj223 over ssh.

I tried the jupyter-server command as you have it and it didn't return an output. Furthermore I didn't find any ports in use. @willprice

[kzinovjev]

I managed to connect to a jupyter server running on a compute node by establishing a regular ssh tunnel from the local machine to the login node and then a reverse one from the compute node to the login node once the job is running.

So, I run this on my laptop:

PORT=`shuf -i 10000-65000 -n 1`

ssh -tt -l <username> -L $PORT:localhost:$PORT jade2.hartree.stfc.ac.uk bash -l << EOF
    utils/jupyter_tunnel.sh $PORT
EOF

It picks a random port, opens a tunnel, and runs the following on the login node (jupyter_tunnel.sh):

#!/bin/bash

PORT=$1
HOST=`hostname`

srun --nodes=1 --gres=gpu:1 --time=12:00:00 --partition=small bash << EOF
module load python/anaconda3

ssh -N -f -R $PORT:localhost:$PORT -o StrictHostKeyChecking=no $HOST
jupyter lab --no-browser --port $PORT
EOF

I also had to generate an ssh key pair (ssh-keygen) on the login node and add my own id_rsa.pub to authorized_keys to make sure the login node accepts ssh connections from the compute nodes.