search

berkeley-dsep-infra / jupyterhub-k8s

[Deprecated] Data 8's deployment of JupyterHub on Kubernetes
Apache License 2.0
34 stars 17 forks source link

Some pods are not killed by cull #116

Closed SaladRaider closed 7 years ago

SaladRaider commented 7 years ago

There are some pods that have suspiciously been alive for a day or more.

Some examples are: jupyter-yitian-zhang-1217 jupyter-t-wong-1186 jupyter-niraj-a-rao-1156

Thank you @tonyyanga for bringing up this issue.

yuvipanda commented 7 years ago

I suspect these accounts may no longer exist - they seem similar to user accounts we deleted during the feb 9 incident. Hence JupyterHub wouldn't know of them, and neither would the culler.

On Sat, Feb 18, 2017 at 8:03 PM, Peter Veerman notifications@github.com wrote:

There are some pods that have suspiciously been alive for a day or more.

Some examples are: jupyter-yitian-zhang-1217 jupyter-t-wong-1186 jupyter-niraj-a-rao-1156

Thank you @tonyyanga https://github.com/tonyyanga for bringing up this issue.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/data-8/jupyterhub-k8s/issues/116, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB23q-E6CHrCdIwtuo9n3IvH1W2PcWnks5rd78JgaJpZM4MFWcI .

-- Yuvi Panda T http://yuvi.in/blog

SaladRaider commented 7 years ago

I see. Ok, I guess this is not a persistent issue.

Perhaps it would be helpful for cull to kill all user pods that are not recognized in the hub database?

yuvipanda commented 7 years ago

I think we should just do it as a one-time thing. 'all user pods that are not recognized' is slightly tricky, since it might accidentally take out the proxy or hub pods, or even the culler itself. And it would also go haywire when strange things happen, such as the hub db being empty for whatever reason. I'd rather have our failure mode be 'extra pods running' rather than 'important pods gone'.

We should just do a run of duplicate user deletion instead.

On Sun, Feb 19, 2017 at 2:04 PM, Peter Veerman notifications@github.com wrote:

I see. Ok, I guess this is not a persistent issue.

Perhaps it would be helpful for cull to kill all user pods that are not recognized in the hub database?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/data-8/jupyterhub-k8s/issues/116#issuecomment-280953180, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB23pGBzIuYMzAur4xU4-aia6oCjBhJks5reLxugaJpZM4MFWcI .

-- Yuvi Panda T http://yuvi.in/blog

yuvipanda commented 7 years ago

I think I've deleted most of these pods. Can someone confirm?

SaladRaider commented 7 years ago

It seems like this issue happens only if something wrong happens with the deployment. Doesn't seem like a persistent issue. I'm going to close this.