search

voila-dashboards / voila

Voilà turns Jupyter notebooks into standalone web applications
https://voila.readthedocs.io
Other
5.4k stars 501 forks source link

Pre-execute notebooks #712

Open davidbrochart opened 4 years ago

davidbrochart commented 4 years ago

In order to hide the start-up time, it would be nice to have the possibility to create a kernel and have it execute the notebook in advance, so that when we enter the URL in the browser, all there is left to do is rendering the HTML and connecting to the kernel. We could have an option such as --pre-execute=nb (default to 0), that would create a pool of already executed notebooks ready to be served, in case there is sudden batch of connections.

maartenbreddels commented 4 years ago

Yes, I think that's a really good thing to have, especially when voila hosts a single notebook. The only requirement is that the notebook is a 'pure function', i.e. not dependant on parameters, time, etc.

SylvainCorlay commented 4 years ago

Also linking

timkpaine commented 4 years ago

Yes, I think that's a really good thing to have, especially when voila hosts a single notebook. The only requirement is that the notebook is a 'pure function', i.e. not dependant on parameters, time, etc.

For those that are not, or when you want to do things like per-user auth inside the notebook once a user has connected, it would be useful to have a few options:

As always, fully preexecuting the notebook should never be the default or only behavior since this makes per-user authenticated notebooks difficult or impossible (we use the details of the user's request to inject auth into the notebook).

davidbrochart commented 4 years ago

partial prexecution (start notebooks, maybe accept a list of modules to import to "warm" the kernel, e.g. things like numpy, pandas, tensorflow, anything else that you might want to setup but without any of the side effects of executing the notebook)

partial pre-execution sounds to me like "execute first N cells", and we would let the author of the notebook arrange it so that the first N cells are user-independent.

timkpaine commented 4 years ago

partial pre-execution sounds to me like "execute first N cells", and we would let the author of the notebook arrange it so that the first N cells are user-independent.

this presupposes that the notebook author and the voila deployer are the same person (or closely in sync), which is not always the case. However with my prelaunch hook, I can always inject cells, so I can implement "warm these modules" with my own custom logic by injecting a cell and saying "warm 1 cell". So tl;dr, I can use it if we say partial is "n cells".

timkpaine commented 3 years ago

starting in https://github.com/voila-dashboards/voila/pull/726, I have a pretty firm idea of how to do it so should be ready shortly. Interested in any thoughts around how to test it, though.

maartenbreddels commented 3 years ago

Nice work Tim, regarding preheat, we could do that based on a cell tag 'voila-preheat' orso.