Ivy-Apps / ivy-wallet

Ivy Wallet is an open-source money manager app for Android, no longer maintained. You can fork the code or download the final version from Google Play.
https://play.google.com/store/apps/details?id=com.ivy.wallet
GNU General Public License v3.0
2.8k stars 675 forks source link

Migrate Paparazzi to Compose Preview Screenshot testing #3507

Closed isaacnguyen0809 closed 1 month ago

isaacnguyen0809 commented 2 months ago

Pull request (PR) checklist

Please check if your pull request fulfills the following requirements:

Description

What's changed?

Risk factors

...

Does this PR close any GitHub issues?

Troubleshooting CI failures ❌

Pull request checks failing? Read our CI Troubleshooting guide.

ILIYANGERMANOV commented 2 months ago

Nice! Google have added threshold support. A few notes:

isaacnguyen0809 commented 2 months ago

@ILIYANGERMANOV - please ignore this comment for now, i need to check it again

According to what I have tried, an image difference threshold of 6% can pass all cases. The case that raises the threshold to 6% is due to the Disclaimer Screen. I will attach the test results in the file below, with a 3% threshold, it passes only 50%. Excluding this screen, I find that the remaining test cases can pass with a threshold of 1.3%.

image

Therefore, i think we have two options:

  1. We accept a 6% threshold (if you’re comfortable with this)
  2. Because the Disclaimer Screen is just a static screen showing text without any complex UI, we can skip the screenshot test for this screen and use the 1.3% threshold for the rest. Wdyt?

BTW, I have updated the docs for this part. Could you check if anything else is needed?

ILIYANGERMANOV commented 2 months ago

Nice work @isaacnguyen0809! I think 2) is a good option. The best choice would be to set 1.3% for all tests and 6% exception for the Disclaimer (e.g. support threshold customization for different tests). Or we can find what makes the Disclaimer screen flaky.

Anyway, I'm happy with skipping its tests if my suggestions are too hard to impl

ILIYANGERMANOV commented 1 month ago

We can re-open it when done