Closed parente closed 8 years ago
/cc @rgbkrk @jhpedemonte @dalogsdon @nitind @lull3rskat3r because we talked about it and @ellisonbg because it's a clear thing for us to try on workbench at some point
One more note: I know I need to do due diligence on looking at if this could be solved with the existing nbconvert support in notebook and plugging in exports to nbconvert itself. It feels a little weird on the face of it to have nbconvert doing things like invoking docker build and pushing an image to a registry or writing a Cloud Foundry manifest alongside a notebook converted to a dashboard or ... But maybe it would work?
nbconvert investigation notes:
The whole config / ConfigManager system will really help here too:
Decided to forgo shimming this into existing notebook nbconvert support for the time being, primarily because the menu item JS is not dynamic and the handler only works for downloads.
The associated PR has a prototype implementation. An extension can register itself for inclusion in the download or deploy menus by:
from notebook.services.config import ConfigManager
cm = ConfigManager()
cm.update('notebook',
{ 'jupyter_cms_bundlers': {
'my_bundle_id': {
'label': 'My menu item label',
'module_name': 'some.installed.python.package',
'group': 'deploy' # or 'download'
}
}
})
This can be done during a setup.py run or after the fact. The some.installed.python.package
must implement a function with the following signature:
def bundle(handler, absolute_notebook_path):
'''
Performs arbitrary bundling options and then responds in an arbitrary manner.
The caller will invoke finish() on the web response after this function returns.
:param handler: The tornado web handler that serviced the request
:param absolute_notebook_path: The path of the notebook on disk
'''
I opened a handful of repos that will be host the refactored deployment / bundling code that currently all resides in jupyter-incubator/dashboards
. These will drive further refinement of the API here. For example, the call to bundle() should probably be a gen.maybe_future
to allow long running ops to be asynchronous.
Definitely like where this is headed, especially with a decent bundle and ship option. :+1:
I started teasing out the deployment code from the dashboards repo into multiple other repos. The separate naming, versioning, and maintenance of all these repo begins to look a bit duanting, even with just the handful we need in the incubator to prove the concept.
KISS should be the rule to start.
Start with two repos, dashboards_bundlers
and kernel_gateway_bundlers
. These will offer just two installable packages, jupyter_dashboards_bundlers
and jupyter_kernel_gateway_bundlers
. Each will install one or more reference implementation bundlers. Over time, these can be split up in to multiple packages and/or repos. At this point, the cost of such a split is unwarranted.
Most importantly, none of this precludes creating bundlers in other packages and repos. The impl here is agnostic.
First implementation of a very simple bundler is at jupyter-incubator/kernel_gateway_bundlers#1.
Added the maybe_future
wrapper and made the bundler handler a tornado coroutine. Updated the README to indicate the bundle function maybe sync or async.
I think the PR should go in soon. The next step of additions will be related to "utilities" that bundlers tend to need. These can be surfaced by the handler instance.
Starting on jupyter-incubator/dashboards_bundlers#1 which will prompt us to provide some common utils here (e.g., associated files bundling).
Initial implementation PR merged. Two initial reference implementations of options using the features (Download As -> Microservice Docker bundle (.zip), Deploy As -> Local Dashboard) are merged in their respective repos. I think this one can get closed with the initial impl in place. We're going to keep refactoring the bundling options out of the dashboard layout extension (see jupyter-incubator/dashboards#151) and will open follow-ons here as needed if we need more out of the bundler API.
The jupyter-incubator/dashboards project currently adds a server handler and menu items for triggering dashboard-specific download and deployment. There's a PR against jupyter-incubator/kernel_gateway is about turning notebooks into microservices. It makes sense for that, too, to have both download and deployment options, but adding Notebook extensions to the kernel_gateway makes little sense. Moreover, deployment is something that needs to be tailored endlessly for specific use cases, cloud providers, on-prem environments, etc.
Instead of having lots of extensions implement both the frontend triggering and a custom notebook server handler, it would be better to implement these once in a generic way and then have much lighter weight extensions do the following:
deploy(notebook_filename)
functionThen a central dispatcher could simply map clicks on deploy / download menu items in the frontend to modules in the backend.
It would be great if the Jupyter Notebook provided this simple extension point. For starters, we can prototype it here in this extension, and use the dashboard / microservice deployment use cases as drivers for plugging into it. For example:
Once we prove out the idea, we can try porting it to the new Jupyter Workbench framework and see if it still makes sense as a plugin that allows other plugins, or if it should be part of the core wokrbench menu bar and Jupyter Notebook backend.