Combining kernel culling with shutdown_no_activity_timeout in a notebook+jupyter_enterprise_gateway environment, notebookapp.py will ignore the activity timeout until the user refreshes something by logging into the GUI

[stevehaertel]

Describe the bug notebook + jupyter_enterprise_gateway environment does not respect shutdown_no_activity_timeout if kernel culling is turned on.

To Reproduce Steps to reproduce the behavior:

1. Set up your notebook config.py file with the important parts:

   c.NotebookApp.shutdown_no_activity_timeout = 120
   c.KernelManager.autorestart = False

2. Start Jupyter and JEG with culling and no_activity_timeout setup, for example, in this way:

   user 19693 18971  0 15:53 ?        00:00:01 /path/to/python3.7 /path/to/jupyter-enterprisegateway --MappingKernelManager.cull_idle_timeout=400 --MappingKernelManager.cull_interval=100 --MappingKernelManager.cull_connected=True --MappingKernelManager.cull_busy=False --log-level=INFO
   user 19751 18971  0 15:53 ?        00:00:01 /path/to/python /path/to/jupyter-notebook --allow-root --notebook-dir=/path/to/notebooks

3. Launch your NB, and start a kernel

4. Logout from the notebook page where you are (I didn't go back to the main page to log out. Not sure if that matters)

5. Wait until your kernel gets culled successfully. (e.g., watch the output of ps -ef | grep jupyter)

Expected behavior After the kernel is culled, the jupyter process should be shut down after one more interval of shutdown_no_activity_timeout, but it is not.

Desktop (please complete the following information):

• OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
• Browser: Firefox
• notebook version 6.3.0 + jupyter_enterprise_gateway version 2.5.0

Additional context

1. This problem has a workaround -- to just log back into the GUI once time, and that triggers a reset of the notebookapp.py's knowledge of how many kernels are running.
2. The polling loop here in notebookapp.py thinks that there are still running kernels until the user refreshes its memory by doing the above-mentioned login:

   def shutdown_no_activity(self):
       """Shutdown server on timeout when there are no kernels or terminals."""
       km = self.kernel_manager
       if len(km) != 0:
           self.log.info("***kernels still running")
           return   # Kernels still running

3. I found this issue which may or may not have been hitting this same problem.

[kevin-bates]

Hi @stevehaertel,

This issue makes sense because kernels culled on the EG server are not detected as culled until their consumer goes to perform another operation. I'd be curious if you see this behavior using lab because lab tends to monitor kernels more closely/frequently than the notebook front-end.

Rather than just check the length of the kernel manager (which is really the MappingKernelManager), I suspect if that was changed to call self.kernel_manager.list_kernels(), then check that result against zero, this may resolve the issue since, when gateway is configured, this will hit the EG server to refresh the list.