search

azavea / noaa-hydro-data

NOAA Phase 2 Hydrological Data Processing
11 stars 3 forks source link

Cull unused Jupyter resources in Kubernetes #65

Closed jpolchlo closed 2 years ago

jpolchlo commented 2 years ago

For our Kubernetes infra, the ultimate goal is to keep only the most limited number of nodes around at all times, with only basic system pods (k8s services, karpenter, and fundamental dask modules) running on a minimally-spec'd (t3.small or t3.medium) core node. We want to scale up for singleuser jupyter notebook pods and dask-related pods only when work is being done.

Inactive notebooks and Dask clusters which were started by inactive or closed Jupyter sessions tend to hang around, which will keep nodes running when no computation is active. It's possible to close these resources down after a grace period. We should be sure to do this to keep the clusters from using excessive resources.

jpolchlo commented 2 years ago

Jupyterhub idle culler will be of use: https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/customizing/user-management.html#culling-user-pods