Return image data to Jupyter front end?

[vsivsi]

This is obviously more of a question / feature request than an issue, but being able to do inline plotting and image display would be a big win for this package!

Maybe this issue can be a starting point for discussion (or feel free to point me to a more appropriate venue, if available).

Here's the problem as I understand it:

1. Jupyter implements a display_data message type that the kernel can use to communicate image data results to the front end. http://jupyter-client.readthedocs.org/en/latest/messaging.html#id3
2. Go has a flexible standard image library, and lots of non-core plotting libraries built on top of it.
3. The "gore-like" REPL that gophernotes uses to execute requests appears to know nothing about images (or anything at all but text produced by the Stringer interface)

Is there any way to bridge the impedance mismatch between 2) and 3)? If we can figure out a way to get image data out of the REPL, it seems pretty straightforward to implement the appropriate display_data messages to communicate it back to the frontend.

In the meantime the only real way to do this is to write images to files and then embed them within markdown cells. But this is suboptimal both because it's clunky and because such images do not update live when the code that generates them changes due to caching, etc.

Anyway, making this work seemlessly would be an incredibly cool enhancement to this already very useful package. I'm pretty new to Go and Jupyter, but a very experienced developer otherwise, so I'm willing to help-out wherever I can.

Thanks!

[dwhitena]

@vsivsi thanks for the info and suggestions! Yes I have been thinking about this as well. It would be amazing to plot things (e.g. from gonum/plot) in the notebook inline.

Do you have any experience with the display_data message? Maybe we can attempt to implement importing of existing images first with this?

[dwhitena]

@vsivsi as far as getting image data out of the REPL, I think this is possible. We should try via the image package: https://golang.org/pkg/image/

[vsivsi]

Hi, no, I haven't implemented any display_data message code. My familiarity with it is only from the bit of reading I did to put this issue together above. There should be Python and Julia code out there in the Jupyter reference that would be instructive, but I haven't gone looking for it.

[dwhitena]

@sbinet here is a relevant issues as we discussed in our emails.

[vsivsi]

Hi @dwhitena !

Has there been any progress on this front? I'm (coincidentally) the JOSS reviewer of your GopherNotes submission, and as I was working on that, I remembered trying out this package a couple years ago and running into this limitation. Looks like it has been extensively updated since those early days, but I still see this open here, which got me wondering....

[dwhitena]

@vsivsi Thanks for following up. The current hope here is to integrate some of the work that the go-hep team is doing with their neugram into here. They are able to display images in the notebook. Basically, we are just waiting on some side hustle time to do the integration.

[t-0-m-1-3]

@dwhitena, I've got side hustle time, I'd love to look into this

[dwhitena]

@t-0-m-1-3 That would be amazing! Just let me know how I can help. Feel free to reach out here or on Gophers slack.

[SpencerPark]

Hi @t-0-m-1-3, I took a stab at it over at SpencerPark/gophernotes/mime-rendering if you want to work from there or even take whatever you can/want and start fresh.

I know the Jupyter protocol well but writing Go API is not yet my strong suit so if you feel it's awkward please replace whatever needs replacing. Also if you have any Jupyter related questions I'm happy to answer them here or over on the Gophers slack.

[cosmos72]

I took a stab at it too, and implemented the Jupyter message 'display_data' for any expression returned by the interpreter that implements Go interface image.Image.

It works nicely :)

See pull request https://github.com/gopherdata/gophernotes/pull/105 and this screenshot:

[cosmos72]

@SpencerPark, I examined your branch https://github.com/SpencerPark/gophernotes/tree/mime-rendering.

It looks very useful, but also a bit complicated to use, since interpreted code needs to create values that implement custom interfaces as github.com/gopherdata/gophernotes/runtime/HTMLRenderable and many others.

Do you mind if I try to simplify it and merge with my branch https://github.com/cosmos72/gophernotes/tree/display_image ?

[cosmos72]

Done, see https://github.com/cosmos72/gophernotes/tree/display_image - I also included it in pull request https://github.com/gopherdata/gophernotes/pull/105 - let me know your opinion on it :)

I inject a display package in the interpreter, containing the functions HTML(string) error, JSON(string) error, JPEG([]byte) error, PNG([]byte) error, SVG(string) error, ...

Example:

display.HTML("<h2>Hello, world!</h2>")

import (
    "net/http"
    "io/ioutil"
)

// image license stated in https://pixabay.com/en/ubuntu-logo-ubuntu-logo-linux-8647/
// CC0 Creative Commons. Free for commercial use. No attribution required
resp, err := http.Get("https://cdn.pixabay.com/photo/2011/08/14/18/11/ubuntu-logo-8647_960_720.png")
bytes, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
display.PNG(bytes)

resp, err := http.Get("https://upload.wikimedia.org/wikipedia/commons/f/fd/Ghostscript_Tiger.svg")
bytes, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
display.SVG(string(bytes))

and the result is

[cosmos72]

Implemented in 43d3181740b401ec284802afbded2855138debda For examples, see examples/Display.ipynb For the discussion that led to the current API, see pull request 105