I'm not sure if this should be classified as a bug of a feature request:
Description
Golden images are stored in /build/paparazzi/failures when there is a regression (next to the delta-*.png files) but, they are not when:
The reference image (in /test/snapshots/images) is not present:
java.lang.AssertionError: File module/ScreenshotTesting/src/test/snapshots/images/...ScreenshotTest_test.png does not exist
at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:60)
at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
The reference image has a different size:
java.lang.IllegalStateException: Images are different sizes
at app.cash.paparazzi.internal.OffByTwo.compare(OffByTwo.kt:11)
at app.cash.paparazzi.internal.ImageUtils.compareImages(ImageUtils.kt:144)
at app.cash.paparazzi.internal.ImageUtils.assertImageSimilar(ImageUtils.kt:55)
at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:73)
at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
Steps to Reproduce
I can create a simple project for that if it is really necessary, but I think the description is fairly simple, and easy to reproduce, without special setup. Please let me know.
Expected behavior
Having golden screenshots generated even when the test fail because it does not match the reference (or if the reference is missing) would be very valuable in CI scenarios, where the newly created golden images could be immediately extracted without having to run an entire pipeline with a new execution of recordPaparazzi Gradle task.
Also, running the recordPaparazzi might lead to unrelated golden images being updated (with invisible variations) that would flood the review diff.
Additional information:
Paparazzi Version: 1.3.5
Note:
In previous versions (pre-OffByTwo), when there was a regression with different image sizes, the delta and golden images were stored to disk on /out/failures.
I'm not sure if this should be classified as a bug of a feature request:
Description Golden images are stored in
/build/paparazzi/failures
when there is a regression (next to thedelta-*.png
files) but, they are not when:The reference image (in
/test/snapshots/images
) is not present:The reference image has a different size:
Steps to Reproduce I can create a simple project for that if it is really necessary, but I think the description is fairly simple, and easy to reproduce, without special setup. Please let me know.
Expected behavior Having golden screenshots generated even when the test fail because it does not match the reference (or if the reference is missing) would be very valuable in CI scenarios, where the newly created golden images could be immediately extracted without having to run an entire pipeline with a new execution of
recordPaparazzi
Gradle task. Also, running therecordPaparazzi
might lead to unrelated golden images being updated (with invisible variations) that would flood the review diff.Additional information:
Note: In previous versions (pre-
OffByTwo
), when there was a regression with different image sizes, the delta and golden images were stored to disk on/out/failures
.