Using the python3 in container for setting up jupyter kernel

[lmanan]

Hello,

I am trying to use the python3 installed in the container (which python gives opt/conda/bin/python) to show up as a kernel in jupyter.

To do so, in the location ${HOME}/.local/share/jupyter/kernels, I specify a new directory called mykernel and added a kernel.json file of the following description:

{
 "language": "python",
 "argv": ["home/lalit/singularity",
   "exec",
   "-w",
   "/home/lalit/singularity/test.simg",
   "/opt/conda/bin/python",
   "-m",
   "ipykernel",
   "-f",
   "{connection_file}"
 ],
 "display_name": "Python 3 (Singularity)"
}

But this gives a kernel not found error (shown below) upon starting a jupyter notebook. I think the paths to the singularity and test.simg are correct and this error stems from some permissions issue. Any clue on how to solve this? Thanks.

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
    return self.result()  # May raise too.
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/handlers.py", line 72, in post
    type=mtype))
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.5/dist-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/manager.py", line 259, in start_kernel
    **kw)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/manager.py", line 204, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_client/launcher.py", line 138, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied

[vsoch]

Hey @MLbyML ! I'll be able to help you further tomorrow (it's the middle of the night here) but one thing that sticks out is that your home path is incorrect, it's missing a leading slash:

• home/lalit/singularity -> /home/lalit/singularity

But actually, if you are using Singularity on sherlock that wouldn't be the path - it should just point to "singularity" which is at /usr/bin/singularity. You are making a lot of assumptions about the user site being on the path, and it's also not clear to me how you are running this (interactively? from the container? what container?) so what I need from you to debug this further is full information to reproduce your case.

• where are you logged in
• show me what's on your path, what modules are loaded
• where did you pull your container from
• any reason to not use on demand?

Thanks! Be back in a few hours.

[vsoch]

I think it's likely those path errors, but another sanity check you can do is to interact with jupyter to list environments. Here is how I re-created your test case on sherlock (but with fixed paths)

mkdir -p /home/users/vsochat/.local/share/jupyter/kernels/mykernel

# Checking my home and singularity locations
echo $HOME
which singularity

# load jupyter (but are you using one in a container?
module load py-jupyter/1.0.0_py36

# list available kernels
$ jupyter kernelspec list
Available kernels:
  python3     /share/software/user/open/py-jupyter/1.0.0_py36/lib/python3.6/site-packages/ipykernel/resources
  mykernel    /home/users/vsochat/.local/share/jupyter/kernels/mykernel

# the jupyter I'm using
$ which jupyter
/share/software/user/open/py-jupyter/1.0.0_py36/bin/jupyter

At this point you'd want to open an interactive (command line) test terminal using it. But I'll stop here because I don't know how to reproduce what you are doing exactly, let me know. I suspect the permissions error just stems from having the wrong path (since the kernels are loaded when you start it up) but it might be something else.

[lmanan]

Hi @vsoch Sorry my post and question were misplaced (I was using a singularity container for running some jobs on my university cluster, but this wasn't related to sherlock). I finally did solve my issue though: thank you very much for the quick replies!