Pluggable Deploy As / Download As options

[parente]

The jupyter-incubator/dashboards project currently adds a server handler and menu items for triggering dashboard-specific download and deployment. There's a PR against jupyter-incubator/kernel_gateway is about turning notebooks into microservices. It makes sense for that, too, to have both download and deployment options, but adding Notebook extensions to the kernel_gateway makes little sense. Moreover, deployment is something that needs to be tailored endlessly for specific use cases, cloud providers, on-prem environments, etc.

Instead of having lots of extensions implement both the frontend triggering and a custom notebook server handler, it would be better to implement these once in a generic way and then have much lighter weight extensions do the following:

1. Declare an ID (e.g., deploy_dashboards_on_mycloud)
2. Declare one or more menu item labels and contexts (e.g., Deploy As -> Dashboard on My Cloud)
3. Declare a Python module with a deploy(notebook_filename) function

Then a central dispatcher could simply map clicks on deploy / download menu items in the frontend to modules in the backend.

It would be great if the Jupyter Notebook provided this simple extension point. For starters, we can prototype it here in this extension, and use the dashboard / microservice deployment use cases as drivers for plugging into it. For example:

• jupyter-incubator/dashboards_download_bundle - Pulls the download option out of jupyter-incubator/dashboards into a new, tiny extension that better separates the dashboard authoring from one type of deployment (currently using Thebe, nbconvert, etc.; later using dashboards_nodejs_app below)
• jupyter-incubator/dashboards_deploy_local - Pulls the option of deploying a dashboard as a static pae within the Jupyter notebook server itself out of jupyter-incubator/dashboards for the same reason as above
• jupyter-incubator/dashboards_nodejs_app - New nodejs dashboard backend that is more secure than the existing thebe-based prototype
• jupyter-incubator/dashboards_deploy_docker - Example deployment mechanism that uses a local docker client to build and deploy the dashboards_nodejs_app to a docker host
• jupyter-incubator/kernel_gateway_download_bundle - New tiny extension that downloads a zip containing the open notebook plus a Dockerfile configured to combine the notebook + kernel kernel gateway for deployment
• etc.

Once we prove out the idea, we can try porting it to the new Jupyter Workbench framework and see if it still makes sense as a plugin that allows other plugins, or if it should be part of the core wokrbench menu bar and Jupyter Notebook backend.

[parente]

/cc @rgbkrk @jhpedemonte @dalogsdon @nitind @lull3rskat3r because we talked about it and @ellisonbg because it's a clear thing for us to try on workbench at some point

[parente]

One more note: I know I need to do due diligence on looking at if this could be solved with the existing nbconvert support in notebook and plugging in exports to nbconvert itself. It feels a little weird on the face of it to have nbconvert doing things like invoking docker build and pushing an image to a registry or writing a Cloud Foundry manifest alongside a notebook converted to a dashboard or ... But maybe it would work?

[parente]

nbconvert investigation notes:

• preprocess, export, postprocess workflow of nbconvert does fit the concept of turning a notebook into something that can be deployed and/or downloaded
• notebook does dynamically look at the nbcovert exporter_map to build the menu of available download options (https://github.com/jupyter/notebook/blob/master/notebook/services/nbconvert/handlers.py#L17) nope: the menu items are hardcoded in the JS (https://github.com/jupyter/notebook/blob/master/notebook/static/notebook/js/menubar.js#L141)
• nbcovert exporter_map / classes do not seem to be extensible in an official way (I've found lots of examples of using custom preprocessors / postprocessors, but not exporters.)
• notebook handlers always return a file from the response; no ability for the conversion to dictate a different type of response (e.g., redirect) other than a zip file instead of the raw file (https://github.com/jupyter/notebook/blob/master/notebook/nbconvert/handlers.py#L108)
• notebook uses whatever nbconvert configuration is specified (c.Exporter) at notebook server launch time and it can't change dynamically thereafter; all logic for convert and deploy would have to be done in the export step (e.g., calling docker, creating a git repo, ...)
• nbconvert's output can contain multiple file resources which the notebook handler can zip up for a bundled response (https://github.com/jupyter/notebook/blob/master/notebook/nbconvert/handlers.py#L34)

[parente]

The whole config / ConfigManager system will really help here too:

• https://github.com/jupyter/notebook/blob/master/notebook/services/config/manager.py
• https://github.com/jupyter/notebook/blob/master/notebook/static/services/config.js

[parente]

Decided to forgo shimming this into existing notebook nbconvert support for the time being, primarily because the menu item JS is not dynamic and the handler only works for downloads.

The associated PR has a prototype implementation. An extension can register itself for inclusion in the download or deploy menus by:

from notebook.services.config import ConfigManager
cm = ConfigManager()
cm.update('notebook', 
  { 'jupyter_cms_bundlers': {
    'my_bundle_id': {
      'label': 'My menu item label',
      'module_name': 'some.installed.python.package',
      'group': 'deploy' # or 'download'
    }
  }
})

This can be done during a setup.py run or after the fact. The some.installed.python.package must implement a function with the following signature:

def bundle(handler, absolute_notebook_path):
  '''
  Performs arbitrary bundling options and then responds in an arbitrary manner.
  The caller will invoke finish() on the web response after this function returns.

  :param handler: The tornado web handler that serviced the request
  :param absolute_notebook_path: The path of the notebook on disk
  ''' 

I opened a handful of repos that will be host the refactored deployment / bundling code that currently all resides in jupyter-incubator/dashboards. These will drive further refinement of the API here. For example, the call to bundle() should probably be a gen.maybe_future to allow long running ops to be asynchronous.

[rgbkrk]

Definitely like where this is headed, especially with a decent bundle and ship option. :+1:

[parente]

I started teasing out the deployment code from the dashboards repo into multiple other repos. The separate naming, versioning, and maintenance of all these repo begins to look a bit duanting, even with just the handful we need in the incubator to prove the concept.

KISS should be the rule to start.

Start with two repos, dashboards_bundlers and kernel_gateway_bundlers. These will offer just two installable packages, jupyter_dashboards_bundlers and jupyter_kernel_gateway_bundlers. Each will install one or more reference implementation bundlers. Over time, these can be split up in to multiple packages and/or repos. At this point, the cost of such a split is unwarranted.

Most importantly, none of this precludes creating bundlers in other packages and repos. The impl here is agnostic.

[parente]

First implementation of a very simple bundler is at jupyter-incubator/kernel_gateway_bundlers#1.

[parente]

Added the maybe_future wrapper and made the bundler handler a tornado coroutine. Updated the README to indicate the bundle function maybe sync or async.

I think the PR should go in soon. The next step of additions will be related to "utilities" that bundlers tend to need. These can be surfaced by the handler instance.

[parente]

Starting on jupyter-incubator/dashboards_bundlers#1 which will prompt us to provide some common utils here (e.g., associated files bundling).

[parente]

Initial implementation PR merged. Two initial reference implementations of options using the features (Download As -> Microservice Docker bundle (.zip), Deploy As -> Local Dashboard) are merged in their respective repos. I think this one can get closed with the initial impl in place. We're going to keep refactoring the bundling options out of the dashboard layout extension (see jupyter-incubator/dashboards#151) and will open follow-ons here as needed if we need more out of the bundler API.