search

jupyterhub / jupyterhub-idle-culler

JupyterHub service to cull idle servers and users
Other
102 stars 37 forks source link

Integrate jupyterhub-idle-culler with KubeIngressProxy #42

Open droctothorpe opened 2 years ago

droctothorpe commented 2 years ago

Proposed change

last_activity data for routes will be available when using configurable-http-proxy as JupyterHub does by default, but if for example traefik-proxy is used as it is in the TLJH distribution, no such data will be available.

We use JupyterHub + KubeIngressProxy + the nginx ingress controller. Based on the docs, I suspect that the last_activity is never updated, so notebooks are culled even when they're active.

Are there any plans to integrate these two capabilities? If not, would you be open to a PR to that effect (assuming it's possible)?

Thanks!

welcome[bot] commented 2 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

manics commented 2 years ago

The idle culler is agnostic to the underlying spawner. Assuming you're right about the cause it sounds like the fix needs to be in KubeIngressProxy not this repo. Is that your understanding? If so I'll transfer the issue to KubeSpawner.

droctothorpe commented 2 years ago

Thanks for getting back to me, @manics!

KIP just creates the ingress resources. The nginx ingress controller is what's actually responsible for managing the proxy, and it's completely oblivious to the concept of last_activity, since that's a very JH / configurable-http-proxy convention.

I guess what I'm wondering if it makes sense to either modify jupyterhub-idle-culler or create a discrete culler that's more generic / isn't contingent on the use of configurable-http-proxy.

Some approaches that come to mind:

  1. Evaluating idleness using a daemon on the notebook container that terminates the container locally instead of relying on a centralized service.
  2. Determining idleness by checking HTTP requests frequency via prometheus.

Totally understand if this is out of scope. We're pretty unique in our adoption of KIP I suspect.

droctothorpe commented 2 years ago

Hm, alternatively, why not hit the /api/status endpoints of the notebooks? I think that's what KF does: https://github.com/kubeflow/kubeflow/blob/master/components/notebook-controller/pkg/culler/culler.go.

manics commented 2 years ago

There are some past discussions on supporting plugins or configurable hooks:

No progress yet, but maybe your use-case could be a driver?

If you haven't already seen https://discourse.jupyter.org/t/which-is-the-correct-way-to-cull-idle-kernels-and-notebook/8123/7 it explains some of the complexity of idle detection?

Did you see https://discourse.jupyter.org/t/jupyterhub-amalthea/12208 (posted today)? Might be worth seeing how they implemented idle culling.