happo / happo.io

Happo is a cross-browser screenshot testing service
https://happo.io
MIT License
197 stars 26 forks source link

Getting Safari diffs after the recent browser update even though the baseline diffs have been updated #267

Closed marcellmueller closed 7 months ago

marcellmueller commented 7 months ago

Recently there was a warning message about a Safari browser update that might cause some diffs though we are still getting them after this with multiple PRs merged to update the baseline. Is this going to go away or should we increase the diff threshold? Thanks for all your hard work on Happo by the way, our team loves using it 🙇

trotzig commented 7 months ago

Hi @marcellmueller! Sorry for the turbulence with the Safari workers. I made some additional changes on Thursday to mitigate this when I found that one of our worker/server machines had a slightly different spec than the others. I ran through some tests and everything looked stable. But when I'm looking into your issue now I realize that we have a stale cache issue. I'll try to explain here, but feel free to ignore if it's TMI. 🙂

Under certain circumstances, Happo can use cached screenshots. The following criteria have to be met:

Pre-rendered HTML is used when using Cypress, Playwright or Happo Examples integration with prerender: true.

In the case of the recent Safari upgrade, we didn't bump the Happo worker version. This meant that caches produced by old workers were in effect.

I'm addressing the caching issue right away. Keep me updated here if you continue to have issues even after merging a new PR to create a new baseline.

marcellmueller commented 7 months ago

Hey @trotzig, really appreciate your looking into this and the explanation of what was happening. Also I hope you didn't work too much on the weekend 😉

This appears to be sorted for us though I will keep monitoring our diffs and let you know if anything else pops up though I think it's safe to close this issue. Thanks again for the help and Happo in general 🙇