Closed isaacnguyen0809 closed 1 month ago
Nice! Google have added threshold support. A few notes:
@ILIYANGERMANOV - please ignore this comment for now, i need to check it again
According to what I have tried, an image difference threshold of 6% can pass all cases. The case that raises the threshold to 6% is due to the Disclaimer Screen. I will attach the test results in the file below, with a 3% threshold, it passes only 50%. Excluding this screen, I find that the remaining test cases can pass with a threshold of 1.3%.
Therefore, i think we have two options:
- We accept a 6% threshold (if you’re comfortable with this)
- Because the Disclaimer Screen is just a static screen showing text without any complex UI, we can skip the screenshot test for this screen and use the 1.3% threshold for the rest. Wdyt?
BTW, I have updated the docs for this part. Could you check if anything else is needed?
Nice work @isaacnguyen0809! I think 2) is a good option. The best choice would be to set 1.3% for all tests and 6% exception for the Disclaimer (e.g. support threshold customization for different tests). Or we can find what makes the Disclaimer screen flaky.
Anyway, I'm happy with skipping its tests if my suggestions are too hard to impl
We can re-open it when done
Pull request (PR) checklist
Please check if your pull request fulfills the following requirements:
main
branch.Description
What's changed?
Risk factors
...
Does this PR close any GitHub issues?
Troubleshooting CI failures ❌
Pull request checks failing? Read our CI Troubleshooting guide.