search

JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

Support anything that can save as PNG #77

Closed oxinabox closed 3 years ago

oxinabox commented 3 years ago

Including plots.

Closes #62 Alternative to #76

cc @juliohm

oxinabox commented 3 years ago

For 1.0 the problem is that it can't resolve it's test-time dependencies. Because 1.0 won't reresolve dependencies at test time. See https://www.oxinabox.net/2021/02/13/Julia-1.6-what-has-changed-since-1.0.html#resolver-willing-to-downgrade-packages-to-install-new-ones-tiered-resolution

For other things, It seems to be because the plot is slightly different. so I guess need to adjust the sensitivity or find a way to make the plot less sensitive to changes.

oxinabox commented 3 years ago

Failure on 1.0 Ubuntu. idk why. They are indeed slightly different

image
juliohm commented 3 years ago

Thank you @oxinabox for pushing this forward ❤️

oxinabox commented 3 years ago

@juliohm do you have any ability to dig into what is going on with the heatmap on ubuntu?

juliohm commented 3 years ago

I am on Manjaro Linux so maybe the issue would reproduce there as well? How can I help? I understand very little about ImageIO and about these possible discrepancies between OSes.

oxinabox commented 3 years ago

I am on Manjaro Linux so maybe the issue would reproduce there as well? How can I help? I understand very little about ImageIO and about these possible discrepancies between OSes.

Can you check out this branch and see if tests pass for you locally?

juliohm commented 3 years ago

I get a couple of warnings but all the tests pass.

image

oxinabox commented 3 years ago

hmmm, Not sure what to do here. Maybe mark it conditionally broken and open an issue? Maybe we can up the tolerance? idk how to up tolerence

juliohm commented 3 years ago

I had the same question in the packages where I am using ReferenceTests.jl How to increase tolerance? The current threshold with PSNR is not very intuitive, at least for me, and guidelines are appreciated. Perhaps increase the default tolerance in ReferenceTests.jl would also be a good idea.

Any thoughts on that @johnnychen94 ? It would be nice to fix this OS-specific issue, perhaps just increasing tolerance for now.

oxinabox commented 3 years ago

Warning: test fails because PSNR -22.433345794677734 < 15 for the heat map

oxinabox commented 3 years ago

OK, I have spun up a linux VM and replicated the failure, not that i can replace it, fixing it should be easier

oxinabox commented 3 years ago

Confirmed it is Plots/GR actually making a clearly different output on linux.

Linux

image

Mac / Windows

image

oxinabox commented 3 years ago

This is https://github.com/JuliaPlots/Plots.jl/issues/2127 Now that i know it is a know bug, I feel ok just disabling this test til it is fixed. It's not our problem to fix.

If there are no objections I will merge this tomorrow