Interactive use with jupyter notebook and docker command

[ghost]

Hello, I found this in the installation section:

python3 -m remote_ikernel manage --add \ --kernel_cmd="singularity run -B /data /path/to/aertslab-pyscenic-0.9.18.sif ipython kernel -f {connection_file}" \ --name="pyscenic-0918-singularity" \ --interface=ssh \ --host=hostname \ --workdir="~/" \ --language=python3

I would like to use docker instead of singularity. I tried to adapt the command line but it is not working. Could you please help me to adapt the command line in order to use jupyter in the scenic docker image? Thank you.

[cflerin]

Hi @rbioinfo ,

I tried to get this working a while back but couldn't quite get there. Our HPC uses Singularity, so the solution you pasted worked well enough for us, and it seems like the equivalent operation in Docker is a bit more complicated. I found a few examples of this though, which might help you (here, and here). It seems that you need to run a script on the remote machine that will parse the kernel connection file, extract the ports, then construct and execute the docker run command using these parameters. If you do solve this, I'd be very interested to see the result though.

[ghost]

Thank you. Maybe it is easier to run jupyter in the docker and then connect to it from the local machine but when I run it inside the pyscenic docker I get this: Exception: Jupyter commandjupyter-notebooknot found

While it works if I create a new docker installing pyscenic using pip.

[cflerin]

Right, the docker image doesn't have jupyterlab installed, only the kernel interface package (jupyter-client). You could build a new image from the existing pySCENIC image and install jupyterlab in it, then it should be possibly to start jupyter notebook from docker (sounds like this is maybe what you've done already).

[vagabond12]

@cflerin

It seems that the "run" should be used with docker, not singularity. And singularity should be used as "singularity exec ...". And you make this error almost in all of your tutorials.

However, I do not understand what is the "connection_file"? And how to configure it? could you help me? Thank you very much!

[cflerin]

Hi @vagabond12 ,

You don't need to change the {connection_file} bit, this is part of remote_ikernel, you can look at their docs for some other examples.

Using singularity run works just fine in this case since we haven't defined an entrypoint in the container. You can of course use exec as well.

[vagabond12]

@cflerin sorry, as singularity run always did not work for me.

The question is that 'Interactive use with jupyter notebook and singularity' does not work for me

(base) [jxx@WHMS ~]$ (base) [jxx@WHMS ~]$ python -m remote_ikernel manage --add --kernel_cmd="singularity run -B /home/jxx/aertslab-pyscenic-0.9.18.sif ipython kernel -f {connection_file}" --name="pyscenic-918-singularity" --interface=ssh --host=WHMS --workdir="/home/jxx" --language=python3 Added kernel ['rik_ssh_WHMS_pyscenic918singularity']: SSH WHMS pyscenic-918-singularity. (base) [jxx@WHMS ~]$

(base) [jxx@WHMS ~]$ jupyter notebook [W 15:06:32.170 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. [I 15:06:32.192 NotebookApp] JupyterLab extension loaded from /home/jxx/anaconda3/lib/python3.7/site-packages/jupyterlab [I 15:06:32.192 NotebookApp] JupyterLab application directory is /home/jxx/anaconda3/share/jupyter/lab [I 15:06:32.193 NotebookApp] Serving notebooks from local directory: /home/jxx [I 15:06:32.193 NotebookApp] The Jupyter Notebook is running at: [I 15:06:32.193 NotebookApp] http://(WHMS or 127.0.0.1):8888/ [I 15:06:32.193 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 15:06:46.050 NotebookApp] Creating new notebook in [I 15:06:46.758 NotebookApp] Kernel started: b2238144-4b01-4261-9b67-3e862b068163 [I 15:06:46.813 remote_ikernel] Remote kernel version: 0.4.6. [I 15:06:46.813 remote_ikernel] File location: /home/jxx/anaconda3/lib/python3.7/site-packages/remote_ikernel/kernel.py. [I 15:06:46.813 remote_ikernel] Launching kernel over SSH. [I 15:06:46.813 remote_ikernel] Login command: 'ssh -o StrictHostKeyChecking=no WHMS'. [I 15:06:46.835 remote_ikernel] Established connection; starting kernel. [I 15:06:46.835 remote_ikernel] Remote working directory /home/jxx. [I 15:06:46.987 remote_ikernel] Running kernel command: 'singularity run -B /home/jxx/aertslab-pyscenic-0.9.18.sif ipython kernel -f ./rik_kernel-b2238144-4b01-4261-9b67-3e862b068163.json'. [I 15:06:47.322 remote_ikernel] Setting up tunnels on ports: 47644, 52387, 50598, 59858, 33913. [E 15:06:52.328 remote_ikernel] Kernel died. [I 15:06:52.760 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports [I 15:06:52.832 remote_ikernel] Remote kernel version: 0.4.6. [I 15:06:52.832 remote_ikernel] File location: /home/jxx/anaconda3/lib/python3.7/site-packages/remote_ikernel/kernel.py. [I 15:06:52.833 remote_ikernel] Launching kernel over SSH. [I 15:06:52.833 remote_ikernel] Login command: 'ssh -o StrictHostKeyChecking=no WHMS'. [I 15:06:52.851 remote_ikernel] Established connection; starting kernel. [I 15:06:52.851 remote_ikernel] Remote working directory /home/jxx. [I 15:06:53.003 remote_ikernel] Running kernel command: 'singularity run -B /home/jxx/aertslab-pyscenic-0.9.18.sif ipython kernel -f ./rik_kernel-b2238144-4b01-4261-9b67-3e862b068163.json'. [I 15:06:53.340 remote_ikernel] Setting up tunnels on ports: 47644, 52387, 50598, 59858, 33913. [E 15:06:58.345 remote_ikernel] Kernel died. [I 15:06:58.776 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports [I 15:06:58.848 remote_ikernel] Remote kernel version: 0.4.6. [I 15:06:58.848 remote_ikernel] File location: /home/jxx/anaconda3/lib/python3.7/site-packages/remote_ikernel/kernel.py. [I 15:06:58.848 remote_ikernel] Launching kernel over SSH. [I 15:06:58.848 remote_ikernel] Login command: 'ssh -o StrictHostKeyChecking=no WHMS'. [I 15:06:58.865 remote_ikernel] Established connection; starting kernel. [I 15:06:58.865 remote_ikernel] Remote working directory /home/jxx. [I 15:06:59.017 remote_ikernel] Running kernel command: 'singularity run -B /home/jxx/aertslab-pyscenic-0.9.18.sif ipython kernel -f ./rik_kernel-b2238144-4b01-4261-9b67-3e862b068163.json'. [I 15:06:59.352 remote_ikernel] Setting up tunnels on ports: 47644, 52387, 50598, 59858, 33913. ^C[I 15:07:01.410 NotebookApp] interrupted Serving notebooks from local directory: /home/jxx 1 active kernel The Jupyter Notebook is running at: http://(WHMS or 127.0.0.1):8888/ Shutdown this notebook server (y/[n])? y [C 15:07:04.042 NotebookApp] Shutdown confirmed [I 15:07:04.044 NotebookApp] Shutting down 1 kernel [E 15:07:04.357 remote_ikernel] Kernel died. [I 15:07:04.647 NotebookApp] Kernel shutdown: b2238144-4b01-4261-9b67-3e862b068163

could you help me? Thank you very much

[cflerin]

@vagabond12 ,

Sorry for the delay. Your command seems to be missing a bind/mount point (-B option). It should look something like:

singularity run -B /path/to/data /home/jxx/aertslab-pyscenic-0.9.18.sif ipython kernel -f {connection_file}

But more generally you can set up either a local or remote kernel by using the following examples. These would go in a kernel file in ~/.local/share/jupyter/kernels/pyscenic-latest/kernel.json (for example).

Remote singularity kernel:

{
  "argv": [
    "/software/jupyter/bin/python",
    "-m",
    "remote_ikernel",
    "--interface",
    "ssh",
    "--host",
    "r23i27n14",
    "--workdir",
    "~/",
    "--kernel_cmd",
    "singularity",
    "exec",
    "-B",
    "/path/to/mounts",
    "/path/to/aertslab-pyscenic-0.11.1.sif",
    "ipython",
    "kernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "pySCENIC singularity remote",
  "language": "Python"
}

Local singularity kernel:

{
    "argv": [
     "singularity",
     "exec",
     "-B",
     "/path/to/mounts",
     "/path/to/aertslab-pyscenic-0.11.1.sif",
     "ipython",
     "kernel",
     "-f",
     "{connection_file}"
    ],
    "display_name": "pySCENIC singularity local",
    "language": "python"
}