RFE: Collapsible JSON tree rendering

[ncoghlan]

I've been doing a fair bit of work with JSON data in notebooks recently, and it would be nice to have an out-of-the-box collapsible tree rendering for JSON-compatible data structures.

The recipe in this Reddit post from @caldwell largely gets the job done: https://www.reddit.com/r/IPython/comments/34t4m7/lpt_print_json_in_collapsible_format_in_ipython/

It's just a bit inconvenient to use at the moment, as you not only need the Python snippet given in the Reddit post, you also need to copy the supporting JavaScript file and its copyright notice around (or else rely on a live internet connection and access to GitHub).

Given enough time I'll figure out how to tidy this up and submit a PR myself, but I figure I'd post the feature request in the meantime in case the possibility intrigues anyone else :)

[Carreau]

Thanks @ncoghlan

I would suggest looking at https://github.com/jupyter/jupyterlab, it's the current effort in rewriting the frontend with a more pluggable API that to define custom handler for various mimetypes. If you want to poke around it's probably better that messing with current notebook sourcecode which is a mess and will be sunset at some point.

Getting this with current notebook UI is feasible, but will likely be a hassle, and one of the eternal questions is which feature of the json render to enable by default :-)

[Carreau]

I guess that could be done in pure html/css by making the "expand" buttons checkboxes. Which is likely to be more correct, and more efficient.

[ncoghlan]

@Carreau Right, most of the logic here is actually in @caldwell's renderjson JavaScript library rather than in the Python code. The only tweaks I've made in my own variant of the notebook snippet are to:

• use a local copy of the JS library rather than the online version
• expand the first level of keys by default

[gnestor]

@ncoghlan I am planning on working on some nbextensions next week to render different mime types and JSON seems like an obvious place to start.

Regarding the Reddit snippet you referenced, if you make the following change:

import uuid
from IPython.display import display_javascript, display_html, display
import json

class RenderJSON(object):
    def __init__(self, json_data):
        if isinstance(json_data, dict):
            self.json_str = json.dumps(json_data)
        else:
            self.json_str = json
        self.uuid = str(uuid.uuid4())

    def _ipython_display_(self):
        display_html('<div id="{}" style="height: 600px; width:100%;"></div>'.format(self.uuid),
            raw=True
        )
        display_javascript("""
        require(["https://rawgit.com/caldwell/renderjson/master/renderjson.js"], function() {
          renderjson.set_show_to_level(1)
          document.getElementById('%s').appendChild(renderjson(%s))
        });
        """ % (self.uuid, self.json_str), raw=True)

specifically the renderjson.set_show_to_level(1), you can expand the first-level key by default.

[jasongrout]

Unless you have a need for the interactive part of widgets (i.e., this data structure will be changing a lot, and you want to keep the output up to date, I think this is better done as a renderer for a specific output mimetype, like @gnestor mentions. Making a JupyterLab plugin would be a great way to deal with it in JupyterLab.

[ncoghlan]

@gnestor Aye, I think your variant here is pretty much the same tweak I made in my own copy of the snippet.

The main thing I don't know is what options are available to make this something that IPython supports via an import rather than needing to copy custom JS files and Python snippets around. Presumably that would involve:

• adding the renderjson JavaScript code (or a functional equivalent) somewhere
• adding the equivalent of the RenderJSON Python code somewhere (perhaps wrapped in a helper function rather than exposed directly)

If there's an established idiom for providing this kind of custom data type rendering, a pointer to a good example of an existing API to emulate would be ideal.

(I posted the RFE to ipywidgets as that's the main library I'm personally familiar with for enhanced JS-integrated output in notebooks, but that doesn't mean it's necessarily the best place for the functionality)

[jasongrout]

The main thing I don't know is what options are available to make this something that IPython supports via an import rather than needing to copy custom JS files and Python snippets around.

While it's possible support one frontend through a python import (e.g., to embed javascript in a python file and send it to the frontend when displaying), that's very brittle and breaks once there is a different frontend. To properly support multiple kinds of frontends intelligently rendering your output in way that makes sense given the capabilities of the frontend, you'll really should separate out the frontend parts from the kernel-side part. You're right that this would be two things: a RenderJSON python function that packages up the output and sends it as an IPython display_data message, with a mimetype perhaps of application/json, and a separate javascript function that renders that data that is installed in the server to be served up to browsers.

We almost have a good example template for this usecase for JupyterLab (we have a bunch of the pieces, just need to put them together). I think the current notebook is more complicated to extend this way, but it sounds like @gnestor knows what to do already.

[gnestor]

I just published an example jupyterlab extension for rendering both JSON files and output with mime-type "application/json" as a tree: https://github.com/gnestor/jupyterlab_json

I have only tested this against the master branch of jupyterlab (0.7.0) so if you try to install it against a released version of jupyterlab (installed via pip), be sure to upgrade to 0.7.0 first.

I'm working on an extension for classic notebook now... I should have something to show early next week.

[jasongrout]

Great!

We should also beef up the rendermime system so that you can claim any mimetype that ends in +json. We discussed some ideas of how to do this in the dev meeting.

[gnestor]

It looks like jupyterlab_vega is already doing this? https://github.com/altair-viz/jupyterlab_vega/blob/master/src/plugin.ts#L50-L51

I've seen this in nteract too: https://github.com/nteract/nteract/blob/0f14082c6e98bbc86a54ea1d3c33d655b1f252f2/src/notebook/components/transforms/plotly.js#L11

[jasongrout]

We can already do what those two do - register specific mime type strings. I'm talking about something more general - handling any mimetype that ends in +json.

[gnestor]

Ok, so if a mime type includes "json" or "+json", then try to render with a matching renderer and if none exists, fall back to the JSON renderer? Like wildcard/regex mime type renderers? I'd be interested in helping with this 👍

[gnestor]

@jasongrout Maybe you can help...I'm working on an nbextension that will render output with mime-type "application/json". My approach thus far is to override the OutputArea class by adding the JSON mime type to OutputArea.output_types, etc. and adding append_json to `OutputArea.append_map. However, it appears that the notebooks and cells/outputs are loaded before this overriding occurs, so overriding this class does not affect the instances of it. How would you go about doing this?

I created an issue on my repo: https://github.com/gnestor/notebook_json/issues/1

[jasongrout]

As discussed above, I think an extension rendering json for the user is more properly done as a classic notebook extension or a plugin in jupyterlab, both of which are apparently in Grant's repo noted above. Let's move the discussion to his repo: https://github.com/gnestor/notebook_json