search

JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

[BUG] Tests started to fail without modification on user side #88

Closed juliohm closed 3 years ago

juliohm commented 3 years ago

Here is the stack trace I am getting running the tests:

Data: Error During Test at /home/juliohm/.julia/dev/Meshes/test/traits.jl:47
  Got exception outside of a @test
  ImageIO writer error: neither save nor fileio_save is defined
    due to MethodError(ImageIO.save, (FileIO.File{FileIO.DataFormat{:PNG}, String}("/tmp/jl_BpAekT/inconversion.png"), Plot{Plots.GRBackend() n=2}), 0x0000000000007424)
    Will try next loader.

  Stacktrace:
    [1] #invokelatest#2
      @ ./essentials.jl:708 [inlined]
    [2] invokelatest
      @ ./essentials.jl:706 [inlined]
    [3] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, file::FileIO.Formatted, args::Plots.Plot{Plots.GRBackend}; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ FileIO ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:212
    [4] action
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:192 [inlined]
    [5] #save#21
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:135 [inlined]
    [6] save
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:132 [inlined]
    [7] savefile
      @ ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:20 [inlined]
    [8] (::ReferenceTests.var"#10#11"{Plots.Plot{Plots.GRBackend}})(dir::String)
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:71
    [9] mktempdir(fn::ReferenceTests.var"#10#11"{Plots.Plot{Plots.GRBackend}}, parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:729
   [10] mktempdir (repeats 2 times)
      @ ./file.jl:727 [inlined]
   [11] _convert(#unused#::Type{FileIO.DataFormat{:PNG}}, data::Plots.Plot{Plots.GRBackend})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:69
   [12] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:PNG}, String}, raw_actual::Plots.Plot{Plots.GRBackend}, equiv::Nothing, rendermode::Nothing; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:103
   [13] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:PNG}, String}, raw_actual::Plots.Plot{Plots.GRBackend}, equiv::Nothing, rendermode::Nothing)
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:100
   [14] test_reference(filename::String, raw_actual::Plots.Plot{Plots.GRBackend}; by::Nothing, render::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:90
   [15] test_reference(filename::String, raw_actual::Plots.Plot{Plots.GRBackend})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:90
   [16] macro expansion
      @ ~/.julia/dev/Meshes/test/traits.jl:129 [inlined]
   [17] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [18] macro expansion
      @ ~/.julia/dev/Meshes/test/traits.jl:49 [inlined]
   [19] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [20] top-level scope
      @ ~/.julia/dev/Meshes/test/traits.jl:2
   [21] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [22] macro expansion
      @ ~/.julia/dev/Meshes/test/runtests.jl:84 [inlined]
   [23] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [24] top-level scope
      @ ~/.julia/dev/Meshes/test/runtests.jl:82
   [25] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [26] top-level scope
      @ none:6
   [27] eval
      @ ./boot.jl:360 [inlined]
   [28] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [29] _start()
      @ Base ./client.jl:485
  Stacktrace:
    [1] handle_error(e::FileIO.WriterError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
      @ FileIO ~/.julia/packages/FileIO/H5jHM/src/error_handling.jl:61
    [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
      @ FileIO ~/.julia/packages/FileIO/H5jHM/src/error_handling.jl:56
    [3] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, file::FileIO.Formatted, args::Plots.Plot{Plots.GRBackend}; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ FileIO ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:221
    [4] action
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:192 [inlined]
    [5] #save#21
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:135 [inlined]
    [6] save
      @ ~/.julia/packages/FileIO/H5jHM/src/loadsave.jl:132 [inlined]
    [7] savefile
      @ ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:20 [inlined]
    [8] (::ReferenceTests.var"#10#11"{Plots.Plot{Plots.GRBackend}})(dir::String)
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:71
    [9] mktempdir(fn::ReferenceTests.var"#10#11"{Plots.Plot{Plots.GRBackend}}, parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:729
   [10] mktempdir (repeats 2 times)
      @ ./file.jl:727 [inlined]
   [11] _convert(#unused#::Type{FileIO.DataFormat{:PNG}}, data::Plots.Plot{Plots.GRBackend})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/fileio.jl:69
   [12] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:PNG}, String}, raw_actual::Plots.Plot{Plots.GRBackend}, equiv::Nothing, rendermode::Nothing; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:103
   [13] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:PNG}, String}, raw_actual::Plots.Plot{Plots.GRBackend}, equiv::Nothing, rendermode::Nothing)
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:100
   [14] test_reference(filename::String, raw_actual::Plots.Plot{Plots.GRBackend}; by::Nothing, render::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:90
   [15] test_reference(filename::String, raw_actual::Plots.Plot{Plots.GRBackend})
      @ ReferenceTests ~/.julia/packages/ReferenceTests/D8LOl/src/test_reference.jl:90
   [16] macro expansion
      @ ~/.julia/dev/Meshes/test/traits.jl:129 [inlined]
   [17] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [18] macro expansion
      @ ~/.julia/dev/Meshes/test/traits.jl:49 [inlined]
   [19] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [20] top-level scope
      @ ~/.julia/dev/Meshes/test/traits.jl:2
   [21] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [22] macro expansion
      @ ~/.julia/dev/Meshes/test/runtests.jl:84 [inlined]
   [23] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [24] top-level scope
      @ ~/.julia/dev/Meshes/test/runtests.jl:82
   [25] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [26] top-level scope
      @ none:6
   [27] eval
      @ ./boot.jl:360 [inlined]
   [28] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [29] _start()
      @ Base ./client.jl:485

Did something change recently that broke the interaction with visual tests and plots?

oxinabox commented 3 years ago

We added a thing so it always creates a file on failure so you can view it. That was the most recent PR merged.

Can you create a minimal verifiable example? (You can pin to the previous release while we debug it, and to see if it's ReferenceTests or the package that is saving the file.)

juliohm commented 3 years ago

The test suite in Meshes.jl is an example, or any other package in JuliaEarth (e.g. GeoStatsBase.jl, Variography.jl)

I can try to find the time to create a MWE, but this issue should be affecting any test suite with visual tests with plots?

juliohm commented 3 years ago

The issue is gone. Thanks for fixing it.

johnnychen94 commented 3 years ago

It's probably some outdated packages in your setup that bring a version incompatibility wrt FileIO v1.6. (FileIO v1.6 is, unfortunately, breaking somehow, see also https://github.com/JuliaIO/FileIO.jl/pull/309)

Without much information, this is just a guess.