search

eBay / flutter_glove_box

Various eBay tools for Flutter development
BSD 3-Clause "New" or "Revised" License
323 stars 69 forks source link

Golden tests occassionally failing with 0.00% diff #160

Closed guy-plentific closed 2 years ago

guy-plentific commented 2 years ago

I am posting this in the hope that someone else has experienced this issue as has an idea of what could be causing it. For apparently no reason, sometimes one or two of our golden tests fail with the error

Pixel test failed, 0.00% diff

I have noticed a similar issue here: https://github.com/flutter/devtools/issues/2935

We have ruled out obvious causes (different versions of Flutter, different operating systems etc.) but are hoping that someone else has experienced something similar and can help us.

coreysprague commented 2 years ago

@guy-plentific is there any pattern to the tests that are impacted by this? Our app has about 900 golden tests in total, and we haven't encountered this.

guy-plentific commented 2 years ago

@coreysprague sorry for the late reply Corey - no, unfortunately there is no particular pattern. I realise there is very little for you to go by on here, I just thought I would mention in case you had seen this before or had any ideas.

johannessachse commented 2 years ago

@guy-plentific I can confirm this issue. But I don't have any solution to this.

phipps980316 commented 2 years ago

Hey @guy-plentific, we started seeing this too with golden tests with 0.00% difference, but it seemed as tho there was a very very small difference in the images for some reason > 0.001%.

We couldn't determine the reason for it after checking different flutter versions across our CI runners but we added a workaround to allow the very small difference as it is was very insignificant.

Details of the workaround can be found here: Workaround

guy-plentific commented 2 years ago

Hey @guy-plentific, we started seeing this too with golden tests with 0.00% difference, but it seemed as tho there was a very very small difference in the images for some reason > 0.001%.

We couldn't determine the reason for it after checking different flutter versions across our CI runners but we added a workaround to allow the very small difference as it is was very insignificant.

Details of the workaround can be found here: Workaround

Thanks a lot Jon! Full link address is https://rows.com/blog/post/writing-a-localfilecomparator-with-threshold-for-flutter-golden-tests

w3ggy commented 1 year ago

Same problem after upgrading Flutter from 3.3.10 to 3.7.0.

Any idea why? Temporarily we set the threshold to 1% but it is not a solution I guess.

We see this error only on Bitrise with the same MacOS version. We cannot reproduce it locally. Maybe some additional system settings are handled now by Flutter starting 3.7 like it was with AppleFontSmoothing system setting.

UPDATED:

Looks like the issue is in different render system on Intel and Mac chips: https://github.com/flutter/flutter/issues/111739#issuecomment-1378705858

befirst commented 1 year ago

This: https://github.com/eBay/flutter_glove_box/issues/160#issuecomment-1295031993 doesn't look like the long-term solution to a problem. For example, our team uses different mac machines, and when we generate snapshots they are different for m1/2 and Intel chips. It's impossible for now to use tests even with the workaround described above because it looks like it works only for comparison (flutter test) and not for a generation (flutter test --update-goldens), so after each feature, we have a lot of newly regenerated snapshots