search

JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
83 stars 15 forks source link

Comparing objects with external artifacts #124

Closed juliohm closed 9 months ago

juliohm commented 9 months ago

We use ReferenceTests.jl for visual regression tests with Makie.jl. The problem we face is that every plot consumes memory in the repository of the package. Could we somehow store the artifacts in an external repository and read these artifacts at test time with a given URL?

How could we standardize this process, and avoid inflating Julia packages with test artifacts?

oxinabox commented 9 months ago

Sure you can. You should be able to replace @test_reference "stringtest1.txt" collect(1:20) with @test_reference download("http://example.com/stringtest1.txt") collect(1:20) and everything should just work.

Or you can put them all some where in a tarball and download and extra that at the start of your test script. Possibly using the Artifacts stdlib, or DataDeps.jl I image solution depends a lot on your use case -- Makie probably has a lot more files than average, the average user is right to stick a handful of PNGs in their git.

Regardless. Its beyond the scope of this package to solve this. We do one thing and do it well, and that is comparing outputs to files. Handling distribution of files not something we can do here. So unless you run into difficulties with how to interact with downloaded files from this package I think this is better discussed e.g. on https://discourse.julialang.org/

juliohm commented 9 months ago

Is the suggested solution enough though? Providing a download(url) as the reference file breaks the ReferenceTests.jl workflow, rigth? By workflow I mean that we can easily update the outdated files after deleting them from disk. The next ] test will write the files to disk again. Could this be generalized to pushing a new file to remote repository instead?