kernel provisioner `cleanup` method is called on kernel start

Description

I discovered this while developing a custom jupyter kernel provisioner.

My provisioner works on Jupyter Lab but not on Voila.

I then discovered that the cleanup method, which by the documentation should be called only as part of the shutdown kernel sequence is actually called on kernel start.

I added in my provisioner a log entry for each function. This is the log of Jupyter Lab

[I 2023-02-21 14:57:52.826 ServerApp] init
[I 2023-02-21 14:57:52.826 ServerApp] pre_launch
[I 2023-02-21 14:57:53.347 ServerApp] launch_kernel
[W 2023-02-21 14:57:53.381 ServerApp] Message signing is disabled.  This is insecure and not recommended!
[I 2023-02-21 14:57:53.381 ServerApp] Kernel started: 5239215c-9ad8-4a14-b2f5-78d95a8073e7
[I 2023-02-21 14:57:56.372 ServerApp] poll
[I 2023-02-21 14:57:59.372 ServerApp] poll

This is the log of Voila:

[Voila] init
[Voila] pre_launch
[Voila] launch_kernel
[Voila] WARNING | Message signing is disabled.  This is insecure and not recommended!
[Voila] Kernel started: f71f8edd-2e1e-4f32-bcf1-b9e51733304f
[Voila] poll
[Voila] ERROR | cleanup False             <<< As you see cleanup is called right after launch_kernel
[Voila] poll
[Voila] poll

I also tried this with the default local_provisioner by adding a log line in the cleanup function (sorry for the ugly debug method :) and I could see it was called as well, so this tends to exclude that my custom provisioner is the cause.

Reproduce

Add the line self.log.error("CLEANUP!!!!!!!") inside function cleanup of local_provisioner.py: https://github.com/jupyter/jupyter_client/blob/adff6b1d4389c885ee7ff4764fc5ffad6fcbe53f/jupyter_client/provisioning/local_provisioner.py#L140-L153
Run Voila on any notebook with a local kernel
The log will show [Voila] ERROR | CLEANUP!!!!!!! right on kernel start

Expected behavior

Not this to happen :)

Context

voila version 0.4.0
Operating System and version: Fedora Linux 37
Browser and version: Firefox 110

Troubleshoot Output

EnvironmentLocationNotFound: Not a conda environment: /usr

CondaEnvException: Unable to determine environment

Please re-run this command with one of the following options:

* Provide an environment name via --name or -n
* Re-run this command inside an activated conda environment.

$PATH:
        /.venv/bin
        /.local/bin
        /bin
        /usr/condabin
        /usr/lib64/ccache
        /usr/local/bin
        /usr/local/sbin
        /usr/bin
        /usr/sbin

sys.path:
        /.venv/bin
        /usr/lib64/python311.zip
        /usr/lib64/python3.11
        /usr/lib64/python3.11/lib-dynload
        /.venv/lib64/python3.11/site-packages
        /.venv/lib/python3.11/site-packages
        

sys.executable:
        /.venv/bin/python

sys.version:
        3.11.1 (main, Jan  6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)]

platform.platform():
        Linux-6.1.11-200.fc37.x86_64-x86_64-with-glibc2.36

which -a jupyter:
        /.venv/bin/jupyter

pip list:
        Package               Version    Editable project location
        --------------------- ---------- ------------------------------------------------
        aiofiles              22.1.0
        aiosqlite             0.18.0
        anyio                 3.6.2
        argon2-cffi           21.3.0
        argon2-cffi-bindings  21.2.0
        arrow                 1.2.3
        asttokens             2.2.1
        attrs                 22.2.0
        Babel                 2.11.0
        backcall              0.2.0
        bcrypt                4.0.1
        beautifulsoup4        4.11.2
        bleach                6.0.0
        certifi               2022.12.7
        cffi                  1.15.1
        charset-normalizer    3.0.1
        comm                  0.1.2
        cryptography          39.0.1
        debugpy               1.6.6
        decorator             5.1.1
        defusedxml            0.7.1
        entrypoints           0.4
        executing             1.2.0
        fastjsonschema        2.16.2
        fqdn                  1.5.1
        gateway_provisioners  0.2.0.dev0
        idna                  3.4
        ipykernel             6.21.2
        ipyremote             0.0.1      
        ipython               8.10.0
        ipython-genutils      0.2.0
        ipywidgets            8.0.4
        isoduration           20.11.0
        jedi                  0.18.2
        Jinja2                3.1.2
        json5                 0.9.11
        jsonpointer           2.3
        jsonschema            4.17.3
        jupyter_client        7.4.1
        jupyter_core          5.2.0
        jupyter-events        0.6.3
        jupyter-server        1.23.6
        jupyter_server_fileid 0.7.0
        jupyter_server_ydoc   0.6.1
        jupyter-ydoc          0.2.2
        jupyterlab            3.6.1
        jupyterlab-pygments   0.2.2
        jupyterlab_server     2.19.0
        jupyterlab-widgets    3.0.5
        MarkupSafe            2.1.2
        matplotlib-inline     0.1.6
        mistune               2.0.5
        nbclassic             0.5.2
        nbclient              0.7.2
        nbconvert             7.2.9
        nbformat              5.7.3
        nest-asyncio          1.5.6
        notebook              6.5.2
        notebook_shim         0.2.2
        overrides             7.3.1
        packaging             23.0
        pandocfilters         1.5.0
        paramiko              3.0.0
        parso                 0.8.3
        pexpect               4.8.0
        pickleshare           0.7.5
        pip                   22.2.2
        platformdirs          3.0.0
        prometheus-client     0.16.0
        prompt-toolkit        3.0.36
        psutil                5.9.4
        ptyprocess            0.7.0
        pure-eval             0.2.2
        pycparser             2.21
        pycryptodomex         3.17
        Pygments              2.14.0
        PyNaCl                1.5.0
        pyrsistent            0.19.3
        python-dateutil       2.8.2
        python-json-logger    2.0.6
        pytz                  2022.7.1
        PyYAML                6.0
        pyzmq                 25.0.0
        requests              2.28.2
        rfc3339-validator     0.1.4
        rfc3986-validator     0.1.1
        Send2Trash            1.8.0
        setuptools            62.6.0
        six                   1.16.0
        sniffio               1.3.0
        soupsieve             2.4
        stack-data            0.6.2
        terminado             0.17.1
        tinycss2              1.2.1
        tornado               6.2
        traitlets             5.9.0
        uri-template          1.2.0
        urllib3               1.26.14
        voila                 0.4.0
        wcwidth               0.2.6
        webcolors             1.12
        webencodings          0.5.1
        websocket-client      1.5.1
        websockets            10.4
        wheel                 0.37.1
        widgetsnbextension    4.0.5
        y-py                  0.5.4
        ypy-websocket         0.8.2

If using JupyterLab

JupyterLab version: 3.6.1

Installed Labextensions

JupyterLab v3.6.1
/.venv/share/jupyter/labextensions
        jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
        @jupyter-widgets/jupyterlab-manager v5.0.5 enabled OK (python, jupyterlab_widgets)
        @voila-dashboards/jupyterlab-preview v2.2.0 enabled OK (python, voila)

My provisioner code in case it is necessary

```python # Copyright Giulio Girardi. All rights reserved import asyncio import re import sys from signal import Signals from typing import Any, Dict, Union from jupyter_client.connect import write_connection_file from jupyter_client.provisioning.provisioner_base import KernelProvisionerBase from paramiko.channel import Channel from paramiko.client import SSHClient from paramiko.common import cMSG_CHANNEL_REQUEST from paramiko.message import Message from traitlets import Unicode class SSHProvisioner(KernelProvisionerBase): host: str = Unicode() # type: ignore client: Union[SSHClient, None] channel: Union[Channel, None] = None ports_cached = True def __init__(self, **kwargs): super().__init__(**kwargs) self.log.info("init") def __del__(self): self.log.info("del") if self.client: self.client.close() async def pre_launch(self, **kwargs: Any) -> Dict[str, Any]: self.log.info("pre_launch") # Get available ports in client self.client = SSHClient() self.client.load_system_host_keys() self.client.connect(self.host) _, stdout, _ = self.client.exec_command( "comm -23 <(seq 10000 65535 | sort) <(ss -Htan | awk '{print $4}' | cut -d':' -f2 | sort -u) | shuf | head -n5", timeout=1, ) shell_port, iopub_port, stdin_port, hb_port, control_port = [ int(line) for line in stdout.readlines() ] file, self.connection_info = write_connection_file( None, shell_port, iopub_port, stdin_port, hb_port, control_port, self.host ) _, stdout, _ = self.client.exec_command("mktemp") self.tempfile = stdout.readline().strip() with self.client.open_sftp() as sftp: sftp.put(file, self.tempfile) # Build the cmd extra_arguments = kwargs.pop("extra_arguments", []) assert self.kernel_spec is not None cmd = self.kernel_spec.argv + extra_arguments ns = dict( connection_file=self.tempfile, prefix=sys.prefix, ) if self.kernel_spec: ns["resource_dir"] = self.kernel_spec.resource_dir ns.update(kwargs) pat = re.compile(r"\{([A-Za-z0-9_]+)\}") def from_ns(match): """Get the key out of ns if it's there, otherwise no change.""" return ns.get(match.group(1), match.group()) kernel_cmd = [pat.sub(from_ns, arg) for arg in cmd] return await super().pre_launch(cmd=kernel_cmd, **kwargs) async def launch_kernel(self, cmd, **kwargs): self.log.info("launch_kernel") stdin, _, _ = self.client.exec_command(" ".join(cmd)) self.channel = stdin.channel return self.connection_info def has_process(self) -> bool: self.log.info("has_process") return self.channel is not None async def poll(self): self.log.info("poll") ret = 0 if self.channel: if self.channel.exit_status_ready(): ret = self.channel.exit_status else: ret = None return ret async def wait(self): self.log.info("wait") ret = 0 if self.channel: while await self.poll() is None: await asyncio.sleep(0.1) ret = await self.poll() return ret async def send_signal(self, signum: int): signame = Signals(signum).name[3:] self.log.info(f"send_signal {signame}") if self.channel: # Until https://github.com/paramiko/paramiko/pull/1535 # gets merged... message = Message() message.add_byte(cMSG_CHANNEL_REQUEST) message.add_int(self.channel.remote_chanid) message.add_string("signal") message.add_boolean(False) message.add_string(signame) self.channel.transport._send_user_message(message) # type: ignore return async def kill(self, restart=False): self.log.info("kill") if self.channel: await self.send_signal(Signals.SIGKILL) async def terminate(self, restart=False): self.log.info("terminate") if self.channel: await self.send_signal(Signals.SIGTERM) async def cleanup(self, restart): self.log.error(f"cleanup {restart}") return # remove this and voila stops working if self.client: self.client.close() self.client = None self.channel = None async def get_provisioner_info(self) -> Dict[str, Any]: self.log.info("get_provisioner_info") return await super().get_provisioner_info() async def load_provisioner_info(self, provisioner_info: Dict) -> None: self.log.info("load_provisioner_info") return await super().load_provisioner_info(provisioner_info) ```

with associated kernelspec

```json { "argv": [ "python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3 (SSH)", "language": "python", "metadata": { "debugger": true, "kernel_provisioner": { "provisioner_name": "ssh-provisioner", "config": { "host": "192.168.1.4" } } } } ```

voila-dashboards / voila