jupyter output: Use mime types for mime bundles instead of publishing everything as `text/html`

[flying-sheep]

Is your feature request related to a problem? Please describe.

I’d like to use multiple backends and/or multiple image formats per backend to create a notebook that displays nicely in different environments.

E.g. GitHub doesn’t show any previews for HoloViews-created plots, probably because it doesn‘t support HTML.

Describe the solution you'd like

1. the display routines need to be changed so images are published as image mime types with metadata instead of HTML. I consider this part a bug. A routine called display_svg has to publish a mime bundle with the key image/svg+xml.
2. there should be a way to have multiple backends active, e.g. matplotlib for static output (that renders on GitHub) and bokeh for interactive frontends.

3. there should be a way to configure backends to render multiple outputs, which already happens for some. E.g. an interactive frontend could publish text/html for compatibility and application/vnd.vegalite.v4+json for frontends that support vegalite4 in a more integrated way.

   but it would also be nice to configure e.g. the matplotlib backend to render both a PNG and a SVG, and leave it to the user’s configured mime preferences to pick which one to display.

Describe alternatives you've considered

Manually hacking together a solution (like pulling the image data out of the HTML payload or manually creating the image payload) is always possible, but cumbersome and shouldn’t be necessary

Additional context

https://nbformat.readthedocs.io/en/latest/format_description.html#display-data

[droumis]

@philippjfr , this is related to the scverse zulip chat

[philippjfr]

Thanks @flying-sheep, we designed some of the mime type stuff a long time ago so I'm using this as a chance to re-familiarize myself with the code. Before I start doing this to describe the current design I will ask some clarifying questions and concerns that cropped up when we first designed this.

I believe we were concerned about always including a bunch of mimetypes in the bundles because it seemed wasteful and expensive to render and then send outputs that wouldn't actually end up getting used. Particularly when rendering Bokeh plots, rendering PNG or SVG outputs is extremely slow and expensive, because it has to launch a headless browser and then orchestrate taking a screenshot with Selenium (or Playwright).

the display routines need to be changed so images are published as image mime types with metadata instead of HTML. I consider this part a bug. A routine called display_svg has to publish a mime bundle with the key image/svg+xml.

I think that should be possible for renderers that support generating SVGs and PNGs cheaply, but I think the reason we are publishing HTML here is for consistency and the ability to center the outputs. I could easily be convinced that centering the outputs was a silly decision and we should just align with everything else but publishing raw SVG/PNG outputs by default still only makes sense when rendering static plots, as soon as you have a HoloMap or DynamicMap, it has to be HTML anyway.

there should be a way to have multiple backends active, e.g. matplotlib for static output (that renders on GitHub) and bokeh for interactive frontends.

We've always wanted to eventually support this but we've never been able to prioritize this in any real sense and the main problem is that each rendering backend has completely separate options, so if you configure the options for bokeh the Matplotlib output may look nothing like it because the options don't translate between the backend. As a general goal this is something we should continue to strive towards.

there should be a way to configure backends to render multiple outputs, which already happens for some. E.g. an interactive frontend could publish text/html for compatibility and application/vnd.vegalite.v4+json for frontends that support vegalite4 in a more integrated way.

Agreed, currently we do something slightly weird and publish the actual content as 'text/html' and then also include an empty 'application/vnd.holoviews_exec.v0+json' value that our Jupyter mime renderers pick up on. This should probably be cleaned up.

But it would also be nice to configure e.g. the matplotlib backend to render both a PNG and a SVG, and leave it to the user’s configured mime preferences to pick which one to display.

This is possible already but as far as I can tell completely undocumented. Specifically you can set:

hv.Store.display_formats = ['png', 'svg', 'html']

When using the bokeh backend this will in fact invoke Selenium to generate a screenshot. Would have to figure out if that's at all desirable before advertising this properly.

[flying-sheep]

Thanks for taking the time to explain all that!

the main problem is that each rendering backend has completely separate options, so if you configure the options for bokeh the Matplotlib output may look nothing like it because the options don't translate between the backend

I guess that means that users should clearly see when they’re configuring backend-specific things and when they configure backend-agnostic things. I thought that was already the case!

hv.Store.display_formats = ['png', 'svg', 'html']

Ideally one would be able to select which backend maps to which mime type. Selenium does sound like overkill, but you do have are lightweight image rendering backends. (I’m aware that the above about options applies)

I think that should be possible for renderers that support generating SVGs and PNGs cheaply, but I think the reason we are publishing HTML here is for consistency and the ability to center the outputs. I could easily be convinced that centering the outputs was a silly decision and we should just align with everything else but publishing raw SVG/PNG outputs by default still only makes sense when rendering static plots, as soon as you have a HoloMap or DynamicMap, it has to be HTML anyway.

I don‘t like that holoviz is publishing a PNG/SVG as HTML instead of as itself. Trust the frontends to know best how they want to display static images. Just add the metadata you have:

{
    "output_type": "execute_result" | "display_data",
    "execution_count": 42,
    "data": {
        "image/png": "[base64-encoded-multiline-png-data]",
        ...
    },
    "metadata": {
        "image/png": {
            "width": 640,
            "height": 480
        }
    }
}