I was writing a fix and test case for
https://github.com/bubkoo/html-to-image/issues/425
when I noticed that despite my fix being stashed all the tests succeeded!
So I assumed maybe my fix wasn't needed after all somehow, but just to be sure I changed the test to be definitely breaking by including the wrong svg; but to my surprise it still succeded even though it printed 404 not found messages in the log.
I then discovered that the compareToRefImage function that most of the tests use compare the two images with a 0.1 threshold and counts it as okay if less than 100 px are different.
Which would be fine if most of the test images were quite large so that 100px would account for small font-rendering and anti aliasing differences (though I would have thought that's what the threshold is for).
But since the test images are quite small, and several of them, such as test/resources/svg-use-tag/image:
which I was basing my new test on, is mostly transparent pixels even when the generated image is completely empty the test still succeeds.
I manually roughly counted the non-transparent pixels of the test image which seems to be around 127px and presumably at least around 60 of these are below the 0.1 threshold to be considered identical to a fully transparent pixel, meaning when I changed thecompareToRefImage to expect less than 1 px not 100px it now failed with only 68px difference from the reference image when generating a completely missing SVG.
Expected Behavior
Failing tests should fail
Current Behavior
Failing tests sometimes succeed
Possible Solution
Rewrite the tests to use much larger test images where the most of the image is non-transparent pixels so that a completely different or missing image cannot accidentally "pass" the test.
Also consider changing that 100px allowed difference to something much smaller too.
But with the current 100px allowance, all the test images and reference images should have at least 1000 non-transparent pixels so that there is no way that a completely missing image could be less than a 100 px difference.
I was writing a fix and test case for https://github.com/bubkoo/html-to-image/issues/425 when I noticed that despite my fix being stashed all the tests succeeded! So I assumed maybe my fix wasn't needed after all somehow, but just to be sure I changed the test to be definitely breaking by including the wrong svg; but to my surprise it still succeded even though it printed 404 not found messages in the log.
I then discovered that the
compareToRefImagefunction that most of the tests use compare the two images with a 0.1 threshold and counts it as okay if less than 100 px are different. Which would be fine if most of the test images were quite large so that 100px would account for small font-rendering and anti aliasing differences (though I would have thought that's what the threshold is for).But since the test images are quite small, and several of them, such as
test/resources/svg-use-tag/image:which I was basing my new test on, is mostly transparent pixels even when the generated image is completely empty the test still succeeds. I manually roughly counted the non-transparent pixels of the test image which seems to be around 127px and presumably at least around 60 of these are below the 0.1 threshold to be considered identical to a fully transparent pixel, meaning when I changed thecompareToRefImage to expect less than 1 px not 100px it now failed with only 68px difference from the reference image when generating a completely missing SVG.
Expected Behavior
Failing tests should fail
Current Behavior
Failing tests sometimes succeed
Possible Solution
Rewrite the tests to use much larger test images where the most of the image is non-transparent pixels so that a completely different or missing image cannot accidentally "pass" the test.
Also consider changing that 100px allowed difference to something much smaller too.
But with the current 100px allowance, all the test images and reference images should have at least 1000 non-transparent pixels so that there is no way that a completely missing image could be less than a 100 px difference.
Steps To Reproduce
Error Message & Stack Trace
```txt ```
Additional Context
Your Environment