generateDiffImages command fails

[sergio-sastre]

Describe the bug I cannot generate Diff images

This issue relates to:

• [x] The Kotlin library
• [x] The Gradle plugin
• [ ] The IntelliJ Platform plugin
• [ ] The sample code
• [ ] The documentation

To Reproduce Steps to reproduce the behavior:

1. Clone this repo
2. Install ImageMagick
3. record baseline for MainActivityScreenshotTest
4. change the text of MainActivityScreenshotTest to "Hello world 2"
5. verify test -> it fails (as expected)
6. run generateDiffImages from the CL and get the following error

Expected behavior It generates the diff

Desktop (please complete the following information):

• OS: MacOs
• Versionc11.6

Target Android Device (please complete the following information):

• Device: Google Pixel 4a
  • Physical or Virtual: Virtual
• API Level: 30
  • Testify Key: 30-1080x2340@440dp-de_DE

[DanielJette]

Hi @sergio-sastre Thanks for the bug report. I'll look into it and try to get a fix included in the next alpha build.

In the meantime, you might want to try out the on-device high-contrast diff capabilities. You might find it suitable for your needs.

Check out this example https://github.com/Shopify/android-testify/blob/41922e18f3a558a77d8118f10a5a9e73648549c9/Sample/src/androidTest/java/com/shopify/testify/sample/ScreenshotRuleExampleTests.kt#L233-L249

TestifyFeatures.GenerateDiffs visualizes exclusion regions and the exactness tolerance. When enabled, GenerateDiffs will write a companion image for your screenshot test which can help you more easily identify which areas of your test have triggered the screenshot failure. Diff files are only generated for failing tests. The generated file will be created in the same directory as your baseline images. Diff files can be pulled from the device using :screenshotPull.

• Black pixels are identical between the baseline and test image
• Grey pixels have been excluded from the comparison
• Yellow pixels are different, but within the Exactness threshold
• Red pixels are different

e.g. https://github.com/Shopify/android-testify/blob/master/Sample/src/androidTest/assets/screenshots/29-1080x2220@440dp-en_US/ScreenshotRuleExampleTests_generateDiffs.diff.png

[alaa-alshaikh]

Hi @sergio-sastre Thanks for the bug report. I'll look into it and try to get a fix included in the next alpha build.

In the meantime, you might want to try out the on-device high-contrast diff capabilities. You might find it suitable for your needs.

Check out this example

https://github.com/Shopify/android-testify/blob/41922e18f3a558a77d8118f10a5a9e73648549c9/Sample/src/androidTest/java/com/shopify/testify/sample/ScreenshotRuleExampleTests.kt#L233-L249

TestifyFeatures.GenerateDiffs visualizes exclusion regions and the exactness tolerance. When enabled, GenerateDiffs will write a companion image for your screenshot test which can help you more easily identify which areas of your test have triggered the screenshot failure. Diff files are only generated for failing tests. The generated file will be created in the same directory as your baseline images. Diff files can be pulled from the device using :screenshotPull.

• Black pixels are identical between the baseline and test image
• Grey pixels have been excluded from the comparison
• Yellow pixels are different, but within the Exactness threshold
• Red pixels are different

e.g. https://github.com/Shopify/android-testify/blob/master/Sample/src/androidTest/assets/screenshots/29-1080x2220@440dp-en_US/ScreenshotRuleExampleTests_generateDiffs.diff.png

@DanielJette I'm getting a black diff screen when I run this, is there a way to fix this?

[DanielJette]

Hmm 🤔 @alaa-alshaikh what is the error/exception being reported by Testify for this test? Is it possible that you have updated your local baseline to match the failing state, thus causing there to be no differences? Is the baseline image recorded exactly as you desired? You'll want to ensure that the baseline image you have in version control is rendered exactly as you intended. Then, the diff image will show you any differences with that image. If they're identical, then you'll get 100% black pixels indicating that the current captured image matches the baseline exactly.

[alaa-alshaikh]

@DanielJette Thank you for your answer, so there is a difference between 2 screenshots, I run compare command from the terminal and I got the below image, but not from testify :(

[DanielJette]

@alaa-alshaikh This could definitely be a bug in the diff algorithm then. Can you provide the following information please?

Desktop:

• OS: [e.g. MacOS, Linux, Windows]
• Version [e.g. 10.15.5, 10]

Target Android Device:

• Device: [e.g. Pixel, S20]
• Physical or Virtual: [e.g. Android Virtual Device (Emulator), Firebase Cloud Image, Samsung Test Lab, Physical Device]
• API Level: [e.g. 21, 28, 30]
• Testify Key: [e.g. Output from ./gradlew testifyKey]

Testify Configuration:

• Testify Version: [e.g. Output from ./gradlew testifyVersion]
• List of enabled TestifyFeatures [e.g. TestifyFeatures.PixelCopyCapture.setEnabled(true)]
• Exactness value: [e.g. The Float passed to rule.setExactness(value)]

[alaa-alshaikh]

@DanielJette

Desktop:

• OS: macOS Big Sur
• Version 11.6

Target Android Device 1:

• Device: Samsung s20+
• Physical or Virtual: Physical
• API Level: 30
• Testify Key : 1080x2400@450dp-en_US

Target Android Device 2:

• Device: Nexus 5X
• Physical or Virtual: Virtual
• API Level: 29
• Testify Key : 29-1080x1920@420dp-en_US

Testify Configuration:

• Testify Version: 1.1.0
• List of enabled TestifyFeatures : only TestifyFeatures.GenerateDiffs.setEnabled(true)
• Exactness value: none

[DanielJette]

Moved to https://github.com/ndtp/android-testify/issues/92