Existing stopped container is not removed when newly requested image is different and `remove = False`

[dijksterhuis]

Bug description

Previously stopped container 1 derived from image A that is different to the latest image B chosen by a user from the options form is not removed when it should be in order to create container 2. Instead, container 1 is simply restarted.

Potential FIx

An additional if block can be added around here:

# if current stopped container's image != image that was just requested
if obj and obj["Config"]["Image"] != image:
    self.log.warning(
        "Removing %s of previously selected image %s that should be cleaned up: %s (id: %s)",
        self.object_type,
        self.image,
        self.object_name,
        self.object_id[:7],
    )
    await self.remove_object()

    obj = None

Can put together a PR. Just tested by editing current master branch install on the host myself and works perfectly well (although jouranlctl doesn't seem to be logging things correctly).

Expected behaviour

A user stopping their server and selecting image B should cause the existing stopped container 1 based on image A to be removed and a new container 2 created based on image B.

Actual behaviour

A user stops their server and selects image B but existing container 1 is restarted.

How to reproduce

With c.Spawner.remove = False and c.Spawner.allowed_images = {image_name_a: image_a, image_name_b: image_b} and a Lab ide.

1. User clicks Start Server and selects image A in the option form
2. User waits for server to load.
3. User goes to File > Hub Control Panel > Stop My Server
4. User clicks Start Server and selects image B in the option form
5. User waits for server to load.
6. User is presented with the container 1 as before (original container 1 is not removed)

Your personal set up

• OS: Ubuntu 18.04
• Version(s): On host:

  $ docker --version && pip3 list | grep jupy && pip3 list | grep docker
  Docker version 20.10.3, build 48d30b5
  jupyter-client                6.1.3
  jupyter-core                  4.6.3
  jupyter-telemetry             0.1.0
  jupyterhub                    1.1.0
  jupyterhub-dummyauthenticator 0.3.1
  jupyterlab                    2.1.3
  jupyterlab-server             1.1.5
  docker                        4.2.0
  dockerspawner                 12.0.0

• Full environment

Container installs (we use custom images available at dockerhub and/or GitHub):

notebook==6.1.5
jupyter==1.0.0
jupyterhub==1.1.0
jupyter_core==4.6.1
jupyterlab==1.2.1
ipython==7.9.0
jupyter-tensorboard
ipykernel==5.0.0

• Configuration

  Output of: `$ grep -v '\(^#\|^[[:space:]]*$\)' /etc/jupyterhub/config.py` ```python import os import requests from jupyter_client.localinterfaces import public_ips from jupyterhub.auth import PAMAuthenticator as Auth SRV = "/srv/jupyterhub/" VAR = "/var/opt/jupyterhub/" CONF = "/etc/jupyterhub" USR = "/usr/local/bin" c = get_config() c.JupyterHub.allow_named_servers = True c.JupyterHub.cookie_secret_file = os.path.join(VAR, 'jupyterhub_cookie_secret') c.JupyterHub.cookie_max_age_days = 60.0 c.JupyterHub.pid_file = os.path.join(VAR, 'jupyterhub.pid') c.JupyterHub.named_server_limit_per_user = 4 c.JupyterHub.admin_access = True c.JupyterHub.db_url = os.path.join('sqlite:/', VAR, 'jupyterhub.sqlite') c.JupyterHub.hub_ip = public_ips()[0] c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jupyterhub/' c.JupyterHub.hub_port = 8080 c.JupyterHub.proxy_api_ip = '0.0.0.0' secret_token = os.urandom(64) c.ConfigurableHTTPProxy.auth_token = secret_token.hex() c.ConfigurableHTTPProxy.pid_file = os.path.join(VAR, "jupyterhub-proxy.pid") c.JupyterHub.logo_file = os.path.abspath(os.path.join(SRV, '.png')) c.JupyterHub.authenticator_class = Auth c.Authenticator.admin_users = {''} c.Authenticator.create_system_users = False c.Authenticator.delete_invalid_users = True c.JupyterHub.spawner_class = 'dockerspawner.SystemUserSpawner' c.Spawner.debug = True c.Spawner.remove = True c.Spawner.name_template = "{username}-{servername}" def get_image_allowlist(Spawner): """ Load the allowlist file and provide the mappings as options to users """ allowlist_file_path = os.path.join(CONF, 'image_allowlist.txt') with open(allowlist_file_path, 'r') as f: data = f.readlines() data = [i.rstrip('\n').split(',') for i in data if i != '\n'] image_allowlist = {k: v.replace(' ', '') for k, v in data} return image_allowlist

c.Spawner.allowed_images = get_image_allowlist

entrypoint.sh will always call start.sh from docker stacks once it's done doing any fancy business

spawn_cmd = os.environ.get('DOCKER_SPAWN_CMD', "entrypoint.sh") c.Spawner.extra_create_kwargs.update({'command': spawn_cmd})

need to mount and chown extra volume/bind mounts

c.Spawner.run_as_root = True c.Spawner.mem_guarantee = 17179869184 c.Spawner.cpu_guarantee = 10.0 c.Spawner.extra_host_config = { 'runtime': 'nvidia', 'pid_mode': 'host', 'shm_size': '16G', } c.Spawner.volumes = { '': '/shares/local/', '': '/shares/network/', }

</details>

- <details><summary>Logs</summary>
<!--
Errors are often logged by jupytehub. How you get logs depends on your deployment.
With kubernetes it might be:

    kubectl get pod # hub pod name starts with hub...
    kubectl logs hub-...
    # or for a single-user server
    kubectl logs jupyter-username

Or the-littlest-jupyterhub:

    journalctl -u jupyterhub
    # or for a single-user server
    journalctl -u jupyter-username
-->
`journalctl -u jupyterhub` since last hub restart:

Mar 29 23:52:31 systemd[1]: Started JupyterHub. Mar 29 23:52:31 jupyterhub[25964]: [W 2021-03-29 23:52:31.947 JupyterHub app:678] JupyterHub.proxy_api_ip is deprecated in JupyterHub 0.8, use ConfigurableHTTPProxy.api_url Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.947 JupyterHub app:2240] Running JupyterHub version 1.1.0 Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.948 JupyterHub app:2271] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.1.0 Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.948 JupyterHub app:2271] Using Spawner: dockerspawner.systemuserspawner.SystemUserSpawner-12.0.0 Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.948 JupyterHub app:2271] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0 Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.955 JupyterHub app:1349] Loading cookie_secret from /var/opt/jupyterhub/jupyterhub_cookie_secret Mar 29 23:52:31 jupyterhub[25964]: [W 2021-03-29 23:52:31.969 JupyterHub configurable:168] Config option delete_invalid_users not recognized by PAMAuthenticator. Mar 29 23:52:31 jupyterhub[25964]: [I 2021-03-29 23:52:31.987 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed. Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.004 JupyterHub app:2311] Initialized 0 spawners in 0.005 seconds Mar 29 23:52:32 jupyterhub[25964]: [W 2021-03-29 23:52:32.006 JupyterHub proxy:643] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else... Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.006 JupyterHub proxy:646] Starting proxy @ http://127.0.0.1:8000/jupyterhub/ Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.405 [ConfigProxy] info: Proxying http://127.0.0.1:8000 to (no default) Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.408 [ConfigProxy] info: Proxy API at http://0.0.0.0:8001/api/routes Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.877 [ConfigProxy] info: 200 GET /api/routes Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.878 JupyterHub app:2556] Hub API listening on http://:8080/jupyterhub/hub/ Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.879 [ConfigProxy] info: 200 GET /api/routes Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.879 JupyterHub proxy:320] Checking routes Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.879 JupyterHub proxy:400] Adding default route for Hub: /jupyterhub/ => http://:8080 Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.880 [ConfigProxy] info: Adding route /jupyterhub -> http://:8080 Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.881 [ConfigProxy] info: Route added /jupyterhub -> http://:8080 Mar 29 23:52:32 jupyterhub[25964]: 23:52:32.881 [ConfigProxy] info: 201 POST /api/routes/jupyterhub Mar 29 23:52:32 jupyterhub[25964]: [I 2021-03-29 23:52:32.882 JupyterHub app:2631] JupyterHub is now running at http://127.0.0.1:8000/jupyterhub/ Mar 29 23:52:36 jupyterhub[25964]: [I 2021-03-29 23:52:36.567 JupyterHub log:174] 200 GET /jupyterhub/hub/home (@) 78.24ms Mar 29 23:52:37 jupyterhub[25964]: [I 2021-03-29 23:52:37.937 JupyterHub log:174] 200 GET /jupyterhub/hub/spawn/ (@) 8.33ms Mar 29 23:52:42 python3[25964]: pam_loginuid(login:session): Error writing /proc/self/loginuid: Operation not permitted Mar 29 23:52:42 python3[25964]: pam_loginuid(login:session): set_loginuid failed Mar 29 23:52:42 python3[25964]: pam_unix(login:session): session opened for user by (uid=0) Mar 29 23:52:42 jupyterhub[25964]: [W 2021-03-29 23:52:42.665 JupyterHub auth:956] Failed to open PAM session for : [PAM Error 14] Cannot make/remove an entry for the specified session Mar 29 23:52:42 jupyterhub[25964]: [W 2021-03-29 23:52:42.666 JupyterHub auth:957] Disabling PAM sessions from now on. Mar 29 23:52:42 jupyterhub[25964]: [I 2021-03-29 23:52:42.837 JupyterHub dockerspawner:942] Container '-' is gone Mar 29 23:52:43 jupyterhub[25964]: [I 2021-03-29 23:52:43.261 JupyterHub dockerspawner:1167] Created container - (id: 8f74cbc) from image Mar 29 23:52:43 jupyterhub[25964]: [I 2021-03-29 23:52:43.261 JupyterHub dockerspawner:1190] Starting container - (id: 8f74cbc)

Second container start doesn't seemed to be caught by logs and/or our server's journalctl isn't updating. 
</details>

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[manics]

There was a discussion on the community forum recently: https://discourse.jupyter.org/t/dockerspawner-container-deletion-by-user-choice/8024/4 Please add your thoughts there since it's a good forum for getting input from the rest of the community.

Setting remove = True should always remove the existing container. The problem with only removing the container if a different image is requested is that a user may not know whether their container will be deleted or not. It's therefore better to always remove the container and document which directories can be used for persistant data.

[dijksterhuis]

Hey @manics hadn't seen that forum post so thanks for the link. Was an interesting read. Will add a new post (and link back here) as it's not exactly the same. Still think that this is weird/unexpected behaviour that should be handled if remove = False is made available as a configuration option.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/dockerspawner-undocumented-behaviour-remove-false-does-not-replace-existing-image-a-with-new-image-b/8576/1

[minrk]

Summarizing what I think we should do from the forum:

1. better document that remove=False is really just for testing and requires manual intervention to delete data
2. warn when using remove=True about pitfalls that it’s not really what should be used
3. persist a user home directory volume by default in a manner that's easily overridden by default
4. switch to remove=True as the default, once home directories are persisted

There is no general 'home directory' persistence that will work in general, but we can pick a default that works for jupyter docker-stacks and make it easy to override. KubeSpawner's been doing this for ages and it works great.