jupyterhub / zero-to-jupyterhub-k8s

Helm Chart & Documentation for deploying JupyterHub on Kubernetes
https://zero-to-jupyterhub.readthedocs.io
Other
1.54k stars 796 forks source link

Clarify documentation about the culling configuration #2244

Closed cdibble closed 2 years ago

cdibble commented 3 years ago

Bug description

There might be more than one thing going on here, so happy to separate into multiple issues if appropriate.

EDIT- there are two things going on indeed: see this issue related to JupyterLab and failing to avoid culling active Lab servers. This issue should focus on idle-culler and the problem of configuring to avoid culling named servers.

I am seeing named servers culled after the timeout interval when that is set to false. This occurs with terminal processes running and the browser/client disconnected (i.e., on overnight jobs).

My config.yaml file used to install the official helm chart (v0.11.1) has the following cull configs:

cull:
  enabled: true
  users: false
  removeNamedServers: false
  timeout: 3600
  every: 600
  concurrency: 10
  maxAge: 0

The hub logs confirm this:

[I 210608 02:28:03 __init__:191] Culling server <user-name>/<server-instance-name> (inactive for 01:05:49)

Expected behaviour

Named servers would not be culled ever.

Servers with running terminal processes would not be culled.

Actual behaviour

After the configured interval, the culler kills the pods.

How to reproduce

See above snippet for the relevant portions of the config.yaml file.

Your personal set up

Running on AWS EKS with helm chart v0.11.1.

The singleUser image uses the base image jupyter/datascience-notebook:python-3.8.8

Other images in helm:

hub:
  image:
    name: jupyterhub/k8s-hub
    tag: 0.10.6-n211.h2e3a73af 

proxy:
  chp:
    image:
      name: jupyterhub/configurable-http-proxy
      tag: 4.3.2
  traefik:
    image:
      name: traefik
      tag: v2.4.8 

singleuser:
  networkTools:
    image:
      name: jupyterhub/k8s-network-tools
      tag: 0.11.1-n221.h84f79652 # "set-by-chartpress"

scheduling:
  userScheduler:
    image:
      name: k8s.gcr.io/kube-scheduler
      tag: v1.19.10

prePuller:
  hook:
    image:
      name: jupyterhub/k8s-image-awaiter
      tag: 0.11.1-n362.h81fdd6e0 # "set-by-chartpress"
  pause:
    image:
      name: k8s.gcr.io/pause
      tag: "3.2"
# paste output of `pip freeze` or `conda list` here
# this is truncated for brevity. can provide more details if needed.
jupyter-client                    6.1.12
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.7.1
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-packaging                 0.10.1
jupyter-resource-usage            0.6.0
jupyter-server                    1.6.4
jupyter-server-mathjax            0.2.2
jupyter-telemetry                 0.1.0
jupyterhub                        1.4.0
jupyterlab                        3.0.14
jupyterlab-git                    0.30.1
jupyterlab-pygments               0.1.2
jupyterlab-server                 2.5.0
jupyterlab-widgets                1.0.0
nbclient                          0.5.3
nbconvert                         6.0.7
nbdime                            3.1.0
nbformat                          5.1.3
nbgitpuller                       0.9.0
notebook                          6.3.0
nteract-on-jupyter                2.1.3
oauthlib                          3.0.1
prometheus-client                 0.10.1
widgetsnbextension                3.5.1

# jupyterhub_config.py