search

rstudio / shinytest

Automated testing for shiny apps
https://rstudio.github.io/shinytest/
Other
225 stars 55 forks source link

Feature request: allow for small differences in cross-platform screenshot comparison #412

Open bersbersbers opened 3 years ago

bersbersbers commented 3 years ago

I usually generate expected test results on a Linux workstation, and tests are failing when run on a Mac due to fonts looking just a tiny bit different. I am aware of #291 being closed, but I wonder if some optional "fuzzy" (= more tolerant) screenshot comparison could be implemented to accept such minor differences.

bersbersbers commented 3 years ago

I should add that vdiffr and gdiff do not seem to feature any such option, but there is https://github.com/MangoTheCat/visualTest at least.

Also, ImageMagick seems to support this use case: https://stackoverflow.com/a/15682469/880783

Edit: I seems I was expecting too much from the PAE - it returns "65535 (1)" for my case - I guess there is at least one pixel which has a black/white inversion. I guess we would need something that takes into account both changes in pixel intensity and distance to the nearest similar pixel, similar to the gamma index in radiotherapy: https://gi-yagit.readthedocs.io/en/latest/gamma_index.html

Using such an index, a pixel in one images passes comparison if its intensity is very close (intensity-wise) to the intensity in the same pixel in the other image, or a pixel with similar intensity is very close (spatially) in the other image.

Closely related: https://github.com/mapbox/pixelmatch, but I need a threshold of close to 1 to make all my pixels pass.