JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

Support for Plots.jl plots? #26

Closed juliohm closed 5 years ago

juliohm commented 5 years ago

First off, thank you for the package :)

I am wondering if there are handlers for plots implemented? I am considering this package instead of VisualRegressionTests.jl because it is well-maintained, and seems more general as well. Could you please elaborate on the dependencies needed for storing plots as images? Can we just use the image handler when we get a plot object?

juliohm commented 5 years ago

@Evizero did you have a chance to take a look at this?

johnnychen94 commented 5 years ago

ReferenceTests uses ImageInTerminal as the backend; https://github.com/Evizero/ReferenceTests.jl/blob/f83e461963799ad642793b1b6ced5f702f16f27e/src/core.jl#L17-L23

I think it would be without much difficults to change to Plots backend as an extension.

juliohm commented 5 years ago

Thank you @johnnychen94 , I am most interested in comparin image plots without relying on Gtk. A preview on the terminal would be perfect.

Do you think it would be possible to add a comparison for Plots.jl plots that compares the images generated directly on the terminal as opposed to launching a Gtk windows for visual comparison? Gtk.jl is quite a dependency. My tests take a while to start just building Gtk.

oxinabox commented 5 years ago

One doesn't need to worry about rendering, that is orthogonal to comparison.

All one needs is a way to convert a plot to a AbstractArray{<:Colorant}.

The following isn't great, it writes a tempfile then reads it back in again. and gives a bunch of warnings. but it works and it gets the idea across.

using Plots, FileIO, Images, ImageMagick, ReferenceTests

function plot2colorant_array(plt)
    mktempdir() do dir
        fn = joinpath(dir, "plot.png")
        save(fn, plt)
        FileIO.load(fn)
    end
end

x = plot2colorant_array(plot(-10:10, (-10:10).^2)
@test_reference "demo.png" x

creates the file

Demo changed

After running the above: running

z = plot2colorant_array(plot(-10:10, rand(21)))
@test_reference "demo.png" z

Output

image

Remember what I said above, the rendering is seperate fron the comparasion. The comparasion is done at full resolution. (usimg some kind of image similarity algorithm from somewhere in the JuliaImages ecosystem).

johnnychen94 commented 5 years ago

I think what @juliohm wants is to show ACTUAL and REFERENCE through Plots, ImageShow or ImageView as an alternative to ImageInTerminal.

ImageInTerminal only shows a blocky thumbnail of the image, there're cases we want to see the high-resolution result.

An example from VisualRegressionTests.jl

image

juliohm commented 5 years ago

Thank you all. The idea is to have the comparison as usual without preview, and then perhaps visualize the images in the terminal when the comparison fails. Is it true that ImageInTerminal.jl can only show thumbnail versions of the images? My main concern with VisualRegressionTests.jl is the Gtk.jl dependency.

oxinabox commented 5 years ago

The idea is to have the comparison as usual without preview, and then perhaps visualize the images in the terminal when the comparison fails.

This is what ReferenceTest.jl does

Is it true that ImageInTerminal.jl can only show thumbnail versions of the images?

Yes, the DPI is fixed by your font size...

Terminals do not have the intrinsic ability to display anothing but unicode+colors. That is all that is standardized. Some terminals (ITerm, kitty) do have extensions for displaying images. I know there have been some experiments in doing that in julia https://github.com/simonschoelly/KittyTerminalImages.jl https://github.com/Keno/TerminalExtensions.jl But they are not standardized, and I don't know much about them.