Add Support for Displaying Matrices as Images

[RoyiAvital]

Any chance to add an option to display a matrix or m x n x c array as image? With the option to set the display size?

A native HTML mode would be great as well.

[j-fu]

Hi,isn't this already covered by Images.jl ? See https://github.com/fonsp/Pluto.jl/issues/60

With Images.jl and Colors.jl you just can do rand(RGB,200,200) and the image is displayed in the notebook. With reinterpret() you can transform the matrix into this format (I think for this it needs to be c x m x n).

Also what do you mean by "native HTML mode" ?

[RoyiAvital]

I'd like support for images in the representation: m x n x c matrix and not only as the type defined in Images.jl. In most other scientific frameworks (MATLAB, Python) images are just matrices in planar layout (In the high level representation).

So supporting this would be great.

By native HTML I meant that the user could chose option to use the img tag in HTML (So behind the scene a binary stream will be created). Also, just an option.

The main motivation for this is able to display images in 1:1 (Pixel for pixel) form.

[j-fu]

I guess the image layout depends on row vs column major storage. I think that c x m x n buffer in Julia has the very same memory layout as the n x m x c buffer in numpy due to the fact that Julia is column major and numpy is row major.

Native html using the img tag is not planned. And I bet one would have convert anyway to c x m x n to make this work, see above. And I guess that this is what images.jl does anyway.

The whole approach here is meant for smoot functions, and I cannot guarantee pixel 1:1.

[RoyiAvital]

I thought the idea is to use JS / HTML for visualization to make things faster (Time to first plot). Hence it makes sense to do so for Images as well (Probably the most used visualization).

Regarding the memory layout, I was under the impression the reason is different. Since it was developed by a team working with images with many channels (I think Tim Holy's team). Their work is mostly on the channels level hence for performance they kept the pixel data contiguous in memory. For classic image processing it is usually a bas choice, performance wise. Usually, in order to utilize SIMD and parallelization, a planar form of the data is preferable. But maybe it was a Python compatibility, I don't know.

Still, the abstraction of image as an mxnxc array is very common.

The img tag has its size properties, so all needed is to have parameters which can be propagated the element. Anyhow, just an idea.

[j-fu]

Hm ImageShow.jl uses the html tag:

https://github.com/JuliaImages/ImageShow.jl/blob/245e0e4fa66001bd4d7b6f51b65b7d46844625cb/src/showmime.jl#L144

But as far as I understand now (my active html knowledge has quite some gaps, I read up on this) this must go throug base64 - which is a big performanc hurt.

So I see the possibility to draw the image directly onto canvas, passing binary data via PlutoRunner.publish_to_js to javascript. I indeed think that this would be faster than the stuff in Images.jl.

Without any transposition, the core API however would have to be 'c x m x n' because that is the way the data is used in canvas - data for each pixel is stored contiguously:

https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/putImageData

Everything else would have to be copied to fit that layout.

So yes in that respect this could fit here. But @fonsp has also some related stuff going on in PlutoUI.jl, this should be coordinated.

[RoyiAvital]

I think the data type is not the important thing here as I can create a small macro or function for my self or even push it later. What's important is the ease of displaying image in 1:1.

Later it would be great to be able to show tool tip per pixel (Value of the pixel).

[j-fu]

When hovering is desired, it makes sense to work via plotly.js:

https://plotly.com/javascript/reference/image/#image

In fact, this appears to make much sense, as the infrastructure for the plotly backend is already here. I think I will go for that. Have some homework on my desk to be done before starting though ...