Servers with activity are getting culled

[yannsartori]

Bug description

If I open a server and exhibit activity on the server (e.g. open a notebook -> create and run cells and periodically save the notebook), it appears that my server gets removed, despite this activity.

Expected behaviour

I would anticipate that my server remains up-and-running.

Actual behaviour

The server gets removed.

How to reproduce

1. Start up a server
2. Edit a notebook, actively saving and running/adding cells
3. See after around --timeout + --cull-every seconds, the server gets culled.

Your personal set up

I am running JupyterHub in Docker with a SwarmSpawner subclass (VariableSwamSpawner) as a spawner class, with support for ContainDSDashboard.

• OS:

  Below is the Dockerfile I use for my jupyterhub server:

  FROM jupyterhub/jupyterhub
  COPY ./certs/ca-bundle.crt /usr/local/share/ca-certificates/
  RUN pip install cdsdashboards oauthenticator dockerspawner ldap3 pyyaml jupyterhub-idle-culler && update-ca-certificates
  COPY ./jupyterhub_config.py /srv/jupyterhub/jupyterhub_config.py
  COPY ./templates /srv/jupyterhub/templates
  # This command may be necesssary if jupyterhub complains about outdated database schemas
  # CMD jupyterhub upgrade-db && jupyterhub

• Version(s):

• Full environment

# pip freeze
alembic @ file:///tmp/wheelhouse/alembic-1.7.5-py3-none-any.whl
async-generator @ file:///tmp/wheelhouse/async_generator-1.10-py3-none-any.whl
attrs @ file:///tmp/wheelhouse/attrs-21.4.0-py2.py3-none-any.whl
cdsdashboards==0.6.0
certifi @ file:///tmp/wheelhouse/certifi-2021.10.8-py2.py3-none-any.whl
certipy @ file:///tmp/wheelhouse/certipy-0.1.3-py3-none-any.whl
cffi @ file:///tmp/wheelhouse/cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
charset-normalizer @ file:///tmp/wheelhouse/charset_normalizer-2.0.10-py3-none-any.whl
cryptography @ file:///tmp/wheelhouse/cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl
docker==5.0.3
dockerspawner==12.1.0
entrypoints @ file:///tmp/wheelhouse/entrypoints-0.3-py2.py3-none-any.whl
escapism==1.0.1
greenlet @ file:///tmp/wheelhouse/greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
idna @ file:///tmp/wheelhouse/idna-3.3-py3-none-any.whl
importlib-metadata @ file:///tmp/wheelhouse/importlib_metadata-4.10.0-py3-none-any.whl
importlib-resources @ file:///tmp/wheelhouse/importlib_resources-5.4.0-py3-none-any.whl
Jinja2 @ file:///tmp/wheelhouse/Jinja2-3.0.3-py3-none-any.whl
jsonschema @ file:///tmp/wheelhouse/jsonschema-4.3.3-py3-none-any.whl
jupyter-telemetry @ file:///tmp/wheelhouse/jupyter_telemetry-0.1.0-py3-none-any.whl
jupyterhub @ file:///tmp/wheelhouse/jupyterhub-2.0.2-py3-none-any.whl
jupyterhub-idle-culler==1.2.1
ldap3==2.9.1
Mako @ file:///tmp/wheelhouse/Mako-1.1.6-py2.py3-none-any.whl
MarkupSafe @ file:///tmp/wheelhouse/MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
oauthenticator==14.2.0
oauthlib @ file:///tmp/wheelhouse/oauthlib-3.1.1-py2.py3-none-any.whl
packaging @ file:///tmp/wheelhouse/packaging-21.3-py3-none-any.whl
pamela @ file:///tmp/wheelhouse/pamela-1.0.0-py2.py3-none-any.whl
pluggy==1.0.0
prometheus-client @ file:///tmp/wheelhouse/prometheus_client-0.12.0-py2.py3-none-any.whl
pyasn1==0.4.8
pycparser @ file:///tmp/wheelhouse/pycparser-2.21-py2.py3-none-any.whl
pycurl==7.43.0.2
pyOpenSSL @ file:///tmp/wheelhouse/pyOpenSSL-21.0.0-py2.py3-none-any.whl
pyparsing @ file:///tmp/wheelhouse/pyparsing-3.0.6-py3-none-any.whl
pyrsistent @ file:///tmp/wheelhouse/pyrsistent-0.18.0-cp38-cp38-manylinux1_x86_64.whl
python-dateutil @ file:///tmp/wheelhouse/python_dateutil-2.8.2-py2.py3-none-any.whl
python-json-logger @ file:///tmp/wheelhouse/python_json_logger-2.0.2-py3-none-any.whl
PyYAML==6.0
requests @ file:///tmp/wheelhouse/requests-2.27.1-py2.py3-none-any.whl
ruamel.yaml @ file:///tmp/wheelhouse/ruamel.yaml-0.17.20-py3-none-any.whl
ruamel.yaml.clib @ file:///tmp/wheelhouse/ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl
six @ file:///tmp/wheelhouse/six-1.16.0-py2.py3-none-any.whl
SQLAlchemy @ file:///tmp/wheelhouse/SQLAlchemy-1.4.29-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tornado @ file:///tmp/wheelhouse/tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl
traitlets @ file:///tmp/wheelhouse/traitlets-5.1.1-py3-none-any.whl
urllib3 @ file:///tmp/wheelhouse/urllib3-1.26.8-py2.py3-none-any.whl
websocket-client==1.2.3
zipp @ file:///tmp/wheelhouse/zipp-3.7.0-py3-none-any.whl

• Configuration

# jupyterhub_config.py
import os
import sys

import yaml
from cdsdashboards.app import CDS_TEMPLATE_PATHS
from cdsdashboards.hubextension import cds_extra_handlers
from ldap3 import Connection, Server

c.Application.log_level = 'DEBUG'

# Configure the user servers
c.JupyterHub.spawner_class = 'cdsdashboards.hubextension.spawners.variabledocker.VariableSwarmSpawner'
c.DockerSpawner.image = os.environ['DOCKER_JUPYTER_IMAGE']
c.Spawner.http_timeout = c.Spawner.start_timeout = int(os.environ['STARTUP_TIMEOUT'])

# Set up default user permissions (the ability to start up others' dashboards
c.JupyterHub.load_roles = [
    {
        'name': 'user',
        'description': 'Standard user privileges',
        'scopes': [
            'self',
            'servers',
            'tokens',
            'list:services',
            'read:services',
            'access:services',
            'access:servers',
        ],
    },
]

# Configure shutting-down/culling
c.JupyterHub.load_roles += [
    {
       "name": "jupyterhub-idle-culler-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
        ],
        # assignment of role's permissions to:
        "services": ["jupyterhub-idle-culler-service"],
    }
]
c.JupyterHub.services = [ # I use extreme values here just to ensure it works
    {
        "name": "jupyterhub-idle-culler-service",
        "command": [
            sys.executable,
            "-m", "jupyterhub_idle_culler",
            "--api-page-size=200",
            "--cull-every=10",
            "--remove-named-servers=True",
            "--timeout=30",
        ],
    }
]

# Configure the network
c.DockerSpawner.network_name = os.environ['DOCKER_NETWORK_NAME']
c.JupyterHub.hub_connect_ip = os.environ['HUB_IP']
# TODO See if this is actually needed
c.DockerSpawner.use_internal_ip = True

# Configure ContainDS
c.JupyterHub.allow_named_servers = True
c.JupyterHub.template_paths = CDS_TEMPLATE_PATHS
c.JupyterHub.extra_handlers = cds_extra_handlers
c.CDSDashboardsConfig.include_servers = True
# ContainDS Auto-culling
c.VariableMixin.proxy_force_alive = False
c.VariableMixin.proxy_last_activity_interval = 15
c.VariableMixin.proxy_request_timeout = 10

c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.dockerbuilder.DockerBuilder'

• Logs