search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
617 stars 222 forks source link

Make EnterpriseGatewayApp.shutdown more robust, since it's a cleanup procedure #1121

Closed CiprianAnton closed 2 years ago

CiprianAnton commented 2 years ago

In our logs, it happened to get a raise web.HTTPError(404, u'Kernel does not exist: %s' % kernel_id) exception in enterprisegatewayapp.shutdown, thus the cleanup being compromised.

Since this function is more a cleanup function, it shouldn't fail and cleanup as much as possible instead. Otherwise, kernels pods may leak.

The "Shut down complete" log is useful to see if termination grace period is enough and EG is not being killed.

rahul26goyal commented 2 years ago

thanks you @CiprianAnton for fixing this.