Open jwortmann opened 4 months ago
I think we're confusing two distinct things: Base.show
and imshow
. imshow
wants a matrix of Colorant
.
Are you thinking of some viewer that can render PNGs directly? For example, are you trying to show this image in a web interface or via VSCode directly?
Are you thinking of some viewer that can render PNGs directly?
Yes. Is ImageView.jl not the right tool for this?
Ideally I only want to implement Base.show
for my type as described in https://docs.julialang.org/en/v1/manual/types/#man-custom-pretty-printing and then the environment where the code is executed decides how to display my type. But for development I would also like to have a simple way to display the image directly from the REPL by opening a window, e.g. via display(d::AbstractDisplay, x)
. So I think I'm searching for a package that provides an AbstractDisplay
for rendering PNG.
I hoped that ImageView.jl could display any custom type directly, if the type can be displayed for example as image/png
etc., but I think that this package doesn't provide an AbstractDisplay
implementation for this, correct?
Or for example something like
function imshow(img::Any)
if showable("image/png", img)
# open a window and render the PNG data returned by Base.show
elseif showable("image/jpeg", img)
# open a window and render the JPEG data returned by Base.show
else
throw(ArgumentError("$(typeof(img)) can't be rendered as an image"))
end
end
I could also imagine that it happens automatically if ImageView is loaded, like
julia> foo = Foo()
# `foo` is displayed as text here
julia> using ImageView
julia> foo
# now `foo` opens as an image in an external window
(or when using just display(foo)
from within a source code file).
But perhaps this would be a bit too intrusive for some users if the new window opens automatically. Therefore having an explicit function like imshow
to render the image is probably the better option.
Typically what you want to do is provide a Matrix{RGB}
or an indexed matrix via IndirectArrays. ImageView will take such a matrix and display it in a Gtk window.
Some displays may understand PNG such as a web browser or a VS Code.
See the Julia images documentation.
While ImageView.jl currently doesn't provide an AbstractDisplay implementation, I saw that ImageInTerminal.jl does support it. However, I believe that its implementation is not entirely correct. It claims that it has support for image/png
for all types which implement a corresponding Base.show
method (https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L7), but in fact it only has specialized implementations for Vector{UInt8}
and for AbstractArray{<:Colorant}
(https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L9-L21).
So when I test the code from above with ImageInTerminal.jl, it still doesn't work:
julia> struct Foo end
julia> foo = Foo()
Foo()
julia> displayable("image/png")
false
julia> using PNGFiles, TestImages, ImageInTerminal
julia> displayable("image/png")
true
julia> showable("image/png", foo)
false
julia> Base.show(io::IO, ::MIME"image/png", ::Foo) = PNGFiles.save(io, testimage("mandrill"))
julia> showable("image/png", foo)
true
julia> foo
Foo()
Therefore, instead of the line
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png") = true
at https://github.com/JuliaImages/ImageInTerminal.jl/blob/340ebebda60b59aa59246938f390c1bfd131958f/src/display.jl#L7, it should actually be
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png") = false
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png", ::Vector{UInt8}) = true
Base.displayable(::TerminalGraphicDisplay, ::MIME"image/png", ::AbstractArray{<:Colorant}) = true
I think there was a misunderstanding in that package about how Base.displayable
is supposed to work. However, it shouldn't be too difficult to provide a proper imlementation with support for image/png
for all types in ImageInTerminal.jl. It just needs to decode the PNG data from Base.show
. Perhaps I can look into that later. Unfortunately even with that, ImageInTerminal wouldn't help me much in practice, because I'm on Windows and the resolution of the displayed images is so low (because it's restricted to use unicode characters and terminal colors on Windows Terminal). So I still hope that ImageView.jl could add support for an AbstractDisplay and open a Gtk window to render the PNG image.
Please also compare how my example code from above works in a Pluto notebook; here everything works as expected:
Update: I created a PR for the ImageInTerminal package at https://github.com/JuliaImages/ImageInTerminal.jl/pull/76
Support for showing PNG's via Base.show
in ImageView may be worth discussing further, but FYI here's a way to show a PNG in a Gtk window using Gtk4.jl:
using Gtk4, TestImages, PNGFiles
struct Foo end
Base.show(io::IO, ::MIME"image/png", x::Foo) = PNGFiles.save(io, testimage("mandrill"))
function displaypng()
io=IOBuffer()
show(io, "image/png", Foo())
b=Gtk4.GLib.GBytes(take!(io))
t=GdkTexture(b)
win=GtkWindow("Image", size(t)[1], size(t)[2]) # set the window dimensions to be the same as the PNG
win[] = GtkPicture(GdkPaintable(t))
end
FYI here's a way to show a PNG in a Gtk window using Gtk4.jl
Thank you for the suggestion. It works well with the example using the file from TestImages, however it turned out that this code randomly crashes (sometimes immediately, or after a few seconds, or sometimes if I resize the window) if it is used with my own implementation of Base.show
. I assume that the bytes or texture object from Gtk4 only uses a reference to the memory where the data is stored, and it expects the user to handle its allocation and deallocation. This is pure speculation though, but I've already experienced a similar problem with SimpleDirectMediaLayer.jl (SDL) which also requires the user to handle the memory management when using certain functions. So I assume that a short time after running the displaypng()
function, the Julia garbage collector cleans up the memory because there isn't any reference anymore to it outside of the function, and then the program crashes due to a memory access violation. A solution could be to permanently keep the data by assigning it to a global variable (this was what I used as a workaround for SDL), but this is maybe a bit awkward.
So I ended up using the approach from CairoMakie, which simply writes the PNG to a temporary file on disk and then it opens the file with the default image viewer:
using Scratch: @get_scratch!
function preview(x::Foo)
filepath = joinpath(@get_scratch!("preview"), "preview.png")
open(filepath, "w") do file
show(file, "image/png", x)
end
command = @static Sys.iswindows() ? `powershell.exe start $filepath` : `open $filepath`
run(command)
nothing
end
OK, thanks for letting me know, I'll look into it -- the memory is supposed to be copied, but maybe there is a reference counting bug somewhere. I'm glad you found an alternative method that works.
How can I use
imshow
to display an image of my custom type which has support for theimage/png
MIME type viaBase.show
?Is it not supported, or is
imshow
not the appropriate function for this? Or does ImageView support some kind of AbstractDisplay, so that I could writedisplay(ImageViewDisplay, Foo())
?Minimum non-working example: