Python kernels in the system prefix without an absolute interpretor path fail to launch if started by `jupyter_client` in a virtual environment

[joouha]

I've received several reports from users where a Python kernel fails to launch when started by a Jupyter client running in a virtual environment (e.g. https://github.com/joouha/euporie/issues/25, https://github.com/joouha/euporie/issues/75), and I've tracked down the cause to these lines of code in jupyter_client.

---

Let's say I have ipykernel installed in my system Python environment, and I want to run this kernel from a client (which uses jupyter_client) installed in a virtual environment:

$ cat /usr/share/jupyter/kernels/python3/kernel.json
{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

$ python -m venv venv

$ ./venv/bin/pip install jupyter-console

$ ./venv/bin/pip uninstall ipykernel 

$ ./venv/bin/jupyter-console --kernel=python3

/home/josiah/venv/bin/python: No module named ipykernel_launcher

jupyter_client replaces the python command in the kernelspec argv line with the path to the interpreter on which jupyter_client is running. This is implemented here:

https://github.com/jupyter/jupyter_client/blob/ed7f9c27b0ae054e2b87d581f6252d240f2e6686/jupyter_client/manager.py#LL273C1-L284C36

I'm really not sure this is expected or desirable behaviour. There is a comment stating the following:

if the current process is in an env and has been launched by abspath without activating the env, python on PATH may not be sys.executable, but it should be.

but I don't think this assumption is correct, and I don't understand what problem was this originally implemented to solve.

[joouha]

This is also relevant:

https://github.com/ipython/ipykernel/blob/f7e5a8fb5c37d81c55b9ea785168c8a19d110eb4/hatch_build.py#L23

When the ipykernel wheel is built, the packaged kernelspec has just python as the first argument of the kernel launch command.

If the user installs the kernel themselves (i.e. python -m ipykernel install --user, the kernelspec is created with an absolute interpreter path.

[fecet]

You have to add envs manually like

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "env":{
     "PATH": PATH after activate venv.
 },
 "display_name": "Python 3.10 (ddip)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

[joouha]

Thanks for your reply, but I think you have misunderstood this issue I described:

I'm writing a new Jupyter client which I want users to run from a virtual environment (using pipx). My client does not depend on ipykernel.

Typically the user will have ipykernel (and a corresponding kernelspec) installed in their system prefix.

When the user tries to launch a kernel from the system kernelspec it fails, because jupyter_client incorrectly edits the kernelspec, substituting the python argv command with the absolute path of the interpreter in the client's venv.

Since ipykernel is not installed in the pipx venv, launching the kernel fails.

[joouha]

I think in the case of a non-absolute python command, cmd[0] should be set to sys._base_executable, unless the client is running in the same prefix that the connection_file is located, in which case cmd[0] could reasonably be set to sys.executable.