Open bjarthur opened 1 week ago
asinghvi17
commented
1 week ago Could it be that your px_per_unit is wrong, and the images are being saved at a higher resolution maybe? I wonder if you get the same results for CairoMakie, which doesn't try to query screen properties.
bjarthur
commented
1 week ago thanks! CairoMakie tests all pass.
curiously though, after running CairoMakie's tests, ReferenceUpdater.serve_update_page_from_dir("CairoMakie/test/reference_images/") still shows no "New images without references" and no "Images with references". and scores_table.tsv is missing:
% tree -L 2 CairoMakie/test/reference_images
CairoMakie/test/reference_images
├── new_files.txt
├── recorded
│ └── CairoMakie
├── reference
│ ├── CairoMakie
│ ├── GLMakie
│ └── WGLMakie
└── scores.tsvif i'm reading the ReferenceUpdater/src/local_servers.jl code correctly, "scores_table.tsv" is only generated by serve_update_page not serve_update_page_from_dir, and without it, no images will ever be displayed.
i could just be using ReferenceUpdater incorrectly. its README seems quite outdated.
ffreyer
commented
6 days ago Reference images usually get desynced temporarily when we merge prs that change or add reference images. The workflow there is:
GLMakie/Heatmap Shader.png
Was caused by that
curiously though, after running CairoMakie's tests, ReferenceUpdater.serve_update_page_from_dir("CairoMakie/test/reference_images/") still shows no "New images without references" and no "Images with references". and scores_table.tsv is missing
That's my bad, I updated things in the wrong place so it only gets generated from a CI download. #4444 should fix that.
bjarthur
commented
6 days ago thanks for the info and the PR @ffreyer
the README says "You should be given a choice of different backend workflow runs. Choose one (usually GLMakie when updating reference images) and confirm". i was never prompted as such. did it used to work this way?
ffreyer
commented
6 days ago That's probably from when we had one set of reference images for all backends.
i'm trying to add a new reference image for https://github.com/MakieOrg/Makie.jl/pull/4293 and having trouble getting the comparison scores to all fall below the threshold when testing locally. in a --temp environment with Makie, MakieCore, GLMakie, and ReferenceTests all dev'd to a 0-day master, i get some variation of the following errors on apple silicon, xvfb-run headless ubuntu 22.04, and windows 10 (both the latter have titan rtx gpus):
error log
``` (jl_rwOy5p) pkg> test GLMakie Testing GLMakie Status `/private/var/folders/s5/8d629n5d7nsf37f60_91wzr40000gq/T/jl_bFliFt/Project.toml`as a result, there is no scores_table.tsv file saved to GLMakie/test/reference_images, and consequently ReferenceUpdater.serve_update_page_from_dir fails to show anything.
how is it that CI currently passes on github? is there a workaround here from me to add the reference images i need for the aforementioned PR? thanks!