Automatically shutdown inactive ipython notebooks

[jeroen]

Something that might be useful would be a way to configure ipython to not keep inactive notebook processes alive indefinitely. For example automatically shutdown the notebook whenever the client disconnects, or after being idle for n minutes.

Context: I would like to use ipython for teaching without setting up individual accounts, or on a public server. I am taking care of security on a lower level using a combination of rlimit and apparmor. If I could configure ipython to shutdown a process once the user disconnects, that makes a cool little sandbox that should be stable enough for teaching.

[pmacaodh]

For my application, an idle timeout feature would be very useful in the context of saving system resources. Working with large datasets, too many idle notebooks can consume the available system memory.

Is there a way currently to poll for notebook kernel age programmatically and shutdown as desired?

[takluyver]

I don't think the server currently records when the kernel was started, but it would be easy enough to add that. I guess what you really want is 'last interaction with the kernel', which might be a bit more complex

[jeroen]

What about shutting down a kernel when the client disconnects, or n seconds after the client has disconnected?

[takluyver]

I'll bring that up in the dev meeting tomorrow - it should be possible to enable that.

[ivanov]

while it would be possible to enable that, I'm not sure it would be desirable to do that...

Disconnecting all clients from a kernel and reconnecting them much later, expecting all of the state to be there is just as valid of a use case as trying to reap zombie kernels in a classroom setting. So I'm :+1: on adding functionality that would make doing something like this possible, but :-1: on enabling it by default or shipping it as a feature, outside of just an example.

[Carreau]

Agreed with @ivanov not to activate it by default.

[takluyver]

I don't think anyone's suggesting activating it by default.

[pmacaodh]

That's right, not by default, but possible somehow.

On 3 August 2014 07:08, Thomas Kluyver notifications@github.com wrote:

I don't think anyone's suggesting activating it by default.

— Reply to this email directly or view it on GitHub https://github.com/ipython/ipython/issues/5539#issuecomment-50982300.

[ellisonbg]

I am fine with something like this (not on by default) but it will be very challenging to come up with a robust and reasonable way of implementing it

Sent from my iPhone

On Aug 2, 2014, at 9:27 AM, Matthias Bussonnier notifications@github.com wrote:

Agreed with @ivanov not to activate it by default.

— Reply to this email directly or view it on GitHub.

[jeroen]

Even the simple case of shutdown notebook on disconnect? A disconnected websocket is usually easy to detect?

[ellisonbg]

The problem is that websockets can be disconnected for a wide range of reasons (spotty wifi). Most of those reasons should not kill the kernel as that action destroys user data. Thus the websocket connections can't be used as the signal for killing the kernel. Because killing a kernel involves data loss we have to err on the side of preservation. We need some signal we can watch that won't ever lead to premature kernel killing.

On Sun, Aug 3, 2014 at 1:15 PM, Jeroen Ooms notifications@github.com wrote:

Even the simple case of shutdown notebook on disconnect? A disconnected websocket is usually easy to detect?

— Reply to this email directly or view it on GitHub https://github.com/ipython/ipython/issues/5539#issuecomment-51001641.

Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[ivanov]

We had a long discussion about this about an hour and a half into this week's dev meeting

[jeroen]

Thanks for the elaborate discussion of this request. Not sure what the final conclusion was, but I hope the "simple" approach can be further explored. A simple option that sets a timer to kill a kernel n seconds after the websocket disconnects (and resets in the event of a reconnect), would be tremendously powerful.

Most websites and web applications use timeouts to prevent sessions from hanging around indefinitely while the user has long forgotten about it. I don't quite understand the argument that timing out sessions is by definition a very complex and application/case specific policy. It seems pretty common sense to me. Sure, the feature can indeed also serve as use case for a potential future generic event-handling policy framework, but that kind of feels like overkill at this point.

[ivanov]

@jeroenooms The point we were trying to make is that "timeout" for some may mean "no client connected" (a kernel can have a web client connected to it via websockets, but it might also have other clients connected to it over zeromq), while others will want it to be "no execution happening" regardless of connections, and to some people sessions hanging around indefinitely is actually a feature. We don't want to put one of these options into IPython, we want to enable others to implement any of them.

[pmacaodh]

Thanks for bringing this up at the latest meeting.

Monitoring kernel events and therefore status through the kernel manager, which can start and stop kernels, would go a long way toward solving the urgent problem in my use case. This sounds like a quick turnaround, so if my opinion counted for anything, I'd side with Thomas' proposal of adding the relatively small set of additional event info while the optimal complete solution is developed. While the admin http API and interfacing with ZeroMQ would be neat, I can't really wait for that.

Does anyone have an example of where I could implement a policy that checks for kernel activity (by subclassing the kernel manager right? which method gets kernel info?). It would be really appreciated.

[takluyver]

There's no example that I know of at present, but I think that you would need to start at IPython.html.services.kernels.handlers.ZMQChannelHandler. The on_message method is called on every incoming message, and it knows its kernel_id. That could schedule a callback to stop the kernel if there's nothing happening for some period of time.

[Carreau]

Why not have another "process" that subscribe to pub/sub and if no "busy" message have come for x-amount of time, send a ask_exit ?

[gebetix]

Hi, guys! Do you have any progress in this issue? I would like to use ipython for teaching too :)

[takluyver]

I think Jupyterhub now has an API that you can talk to to find inactive notebook servers and shut them down. @minrk can tell you more about this. tmpnb also automatically culls inactive notebook servers periodically.

It's still not easy to shut down an individual kernel based on inactivity, but in the things we're currently doing, shutting down a user's notebook server is more useful.

[minrk]

JupyterHub tracks user activity, and this information is made available via a REST API. JupyterHub itself does not cull idle servers or notebooks, but an adjacent service can use the API to cull with whatever logic you would like.

The single-user server does not track activity like the JupyterHub does, so we don't have the information to do similar culling on a per-notebook basis, but we could add it.

[Balajideivam]

Is there any way to implement this? I used the code provided by @minrk but it throws SSL Certificate error.

[minrk]

@Balajideivam can you provide some more details? Is your server running with a self-signed untrusted certificate?

[Balajideivam]

@minrk Yes. Correct. If i remove the SSL and tested it out the same, I am getting HTTP error.

[root@okdev1368 bin]# export JPY_API_TOKEN=jupyterhub token [I 2015-10-20 11:56:49.964 JupyterHub app:619] Not using whitelist. Any authenticated user will be allowed.

[W 151020 11:58:06 iostream:1216] SSL Error on 7 ('10.9.137.168', 8000): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
Traceback (most recent call last):
.........
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

without SSL: I am not passing any parameter to that program. I just updated default values in the .py itself. 

[root@oktst1368 ~]# export JPY_API_TOKEN=`jupyterhub token`
[I 2015-10-20 12:03:18.963 JupyterHub app:654] Not using whitelist. Any authenticated user will be allowed.
[root@oktst1368 ~]# python34 /seamnt/int001/sea/jupyterhub/oauthenticator/src/jupyterhub/examples/cull-idle/cull_idle_servers.py
Traceback (most recent call last):
  File "/seamnt/int001/sea/jupyterhub/oauthenticator/src/jupyterhub/examples/cull-idle/cull_idle_servers.py", line 76, in <module>
    loop.run_sync(cull)
  File "/opt/sea/jupyterhub/anaconda3/lib/python3.4/site-packages/tornado/ioloop.py", line 421, in run_sync
    return future_cell[0].result()
  File "/opt/sea/jupyterhub/anaconda3/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/opt/sea/jupyterhub/anaconda3/lib/python3.4/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/seamnt/int001/sea/jupyterhub/oauthenticator/src/jupyterhub/examples/cull-idle/cull_idle_servers.py", line 43, in cull_idle
    resp = yield client.fetch(req)
  File "/opt/sea/jupyterhub/anaconda3/lib/python3.4/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/sea/jupyterhub/anaconda3/lib/python3.4/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
tornado.httpclient.HTTPError: HTTP 403: Forbidden
[root@oktst1368 ~]#

Could you please help me? 

[SelenaBaset]

Following.

[etejedor]

I had the same issue and I solved it by adding my user to the list of admins, in the JupyterHub configuration file:

c.Authenticator.admin_users = {'youruser'}

And then you do:

export JPY_API_TOKEN=`jupyterhub token youruser`
python cull_idle_servers.py

[chultquist]

Has any progress been made on supporting this for individual kernels? I'm working on a tool that uses jupyter-js-services to spawn Python kernels, and it would be great to have a way to shut these down when the webapp client has been MIA for some time (e.g. a few minutes). I can tell when the webapp is MIA using something like a heartbeat + a Timer thread in Python, so I just need a mechanism on the Python side to actually shut the kernel down. sys.exit() and exit() don't work, and so I'm guessing that Jupyter is doing something clever to prevent users from accidentally killing their kernels.

[chultquist]

FWIW to add to my previous update, I'm able to get somewhere by calling get_ipython().kernel.do_shutdown(False), although then the kernel process ends up as a zombie and so isn't really dead yet. I'm guessing this is because the Jupyter notebook server is keeping a reference to the child.

[takluyver]

There's machinery to cull idle kernels coming in notebook 5.1 - see jupyter/notebook#2215.

[avi-ram]

Following

[JafarShaik6]

Is there anyway to automatic shutdown of notebook through script or config file.

[zhuli19901106]

Following the path of the ancient forebears.