too many open files error

havok2063 commented 4 years ago

After Voila has been running a while, and accessing many kernels, eventually this error is produced and new content is no longer loaded. This occurs

Command to start the Voila server with terminal output showing kernel access

 voila --no-browser --template=embed --enable_nbextensions=True --Voila.tornado_settings="{'allow_origin': 'http://localhost:9000'}" --port=8000
[Voila] Using /var/folders/js/s2zfzbv16019vjh0dft_l_w0000263/T to store connection files
[Voila] Storing connection files in /var/folders/js/s2zfzbv16019vjh0dft_l_w0000263/T/voila_izrrkons.
[Voila] Serving static files from /Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/voila/static.
[Voila] Voila is running at:
http://localhost:8000/
[Voila] Kernel started: 032414d1-d022-410c-8b61-e81bda973534
[Voila] WARNING | Notebook bqplot_vuetify_example.ipynb is not trusted
[Voila] Kernel started: 8350c859-85b8-405b-8f71-f936011184a6
[Voila] WARNING | Notebook notebook2.ipynb is not trusted
[Voila] Kernel started: dc9a26d6-293b-4e3f-8683-785d7988c21c
[Voila] WARNING | Notebook bqplot_vuetify_example.ipynb is not trusted
[Voila] Kernel started: da4310ab-3821-44eb-9858-0b8f29ee091d
[Voila] WARNING | Notebook notebook2.ipynb is not trusted
[Voila] Kernel started: ea8dacb0-1e2a-45d5-b02f-6fa5059060ee
[Voila] Starting buffering for 032414d1-d022-410c-8b61-e81bda973534:646a99ce-5df2-4ad3-8f05-f0d616ddcb2a
[Voila] Kernel started: d9a7b316-9c59-48ca-9050-e176c65d37a1
[Voila] Starting buffering for d9a7b316-9c59-48ca-9050-e176c65d37a1:97decacc-bcff-4096-87c6-bfbb124bf13f
[Voila] Starting buffering for da4310ab-3821-44eb-9858-0b8f29ee091d:ba467d34-4144-4777-ac9b-d93ea5dce1ac
[Voila] Starting buffering for ea8dacb0-1e2a-45d5-b02f-6fa5059060ee:2a1b06c3-59d0-42eb-b770-64059974c57e
[Voila] Restoring connection for da4310ab-3821-44eb-9858-0b8f29ee091d:ba467d34-4144-4777-ac9b-d93ea5dce1ac
[Voila] Restoring connection for ea8dacb0-1e2a-45d5-b02f-6fa5059060ee:2a1b06c3-59d0-42eb-b770-64059974c57e

At some point accessing the content again produces the following error. No new kernel connections can be made after this and content fails to load.

[Voila] Kernel started: 12322707-a045-44e4-85e1-5aed92296b55
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/threading.py", line 932, in _bootstrap_inner
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/channels.py", line 167, in run
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/channels.py", line 94, in _create_socket
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/zmq/sugar/context.py", line 204, in socket
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/zmq/sugar/socket.py", line 59, in __init__
  File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.__init__
zmq.error.ZMQError: Too many open files
[Voila] Starting buffering for 12322707-a045-44e4-85e1-5aed92296b55:0a6bca7b-7600-4273-bfad-984cdbcf6b43
ERROR:tornado.application:Uncaught exception GET /voila/render/asb_example.ipynb (::1)
HTTPServerRequest(protocol='http', host='localhost:8000', method='GET', uri='/voila/render/asb_example.ipynb', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/voila/handler.py", line 101, in get
    self.kernel_id = await tornado.gen.maybe_future(self.kernel_manager.start_kernel(kernel_name=self.notebook.metadata.kernelspec.name, path=self.cwd))
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 170, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/manager.py", line 261, in start_kernel
    self._connect_control_socket()
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/manager.py", line 210, in _connect_control_socket
    self._control_socket = self._create_connected_socket('control')
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/jupyter_client/connect.py", line 618, in _create_connected_socket
    sock = self.context.socket(socket_type)
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/zmq/sugar/context.py", line 204, in socket
    s = self._socket_class(self, socket_type, **kwargs)
  File "/Users/bcherinka/anaconda3/envs/voila_embed/lib/python3.8/site-packages/zmq/sugar/socket.py", line 59, in __init__
    super(Socket, self).__init__(*a, **kw)
  File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.__init__
zmq.error.ZMQError: Too many open files

mariobuikhuizen commented 4 years ago

Thank you for reporting this issue.

A workaround is to run this command in the terminal where Voilà is started: ulimit -n 10000.

I will check if voila-embed is closing kernels correctly and thus closing associated open files.

maartenbreddels commented 4 years ago

but with the ulimit + kernel culling https://voila.readthedocs.io/en/latest/customize.html#cull-idle-kernels it should be better.

havok2063 commented 4 years ago

Ahh thanks for highlighting this. I'll try both of those solutions and see how it goes! Thanks!

mariobuikhuizen / voila-embed

too many open files error #4