search

facebook / screenshot-tests-for-android

Generate fast deterministic screenshots during Android instrumentation tests
http://facebook.github.io/screenshot-tests-for-android
Apache License 2.0
1.74k stars 229 forks source link

RecyclerView screenshot tests are not working on CI #263

Closed VarshaBhatiaPapershift closed 3 years ago

VarshaBhatiaPapershift commented 4 years ago

Hi,

I've been trying to run screenshot tests on CI but getting this below error : 

> Task :design:verifyDebugAndroidTestScreenshotTest
[-m, android_screenshot_tests.pull_screenshots, --apk, /Users/runner/work/Design-Android/Design-Android/design/build/outputs/apk/androidTest/debug/design-debug-androidTest.apk, --temp-dir, /Users/runner/work/Design-Android/Design-Android/design/build/screenshotsDebugAndroidTest, --verify, screenshots]
/sdcard/screenshots/com.papershift.design.test/screenshots-default/metadata.xml: 1 file pulled, 0 skipped. 27.8 MB/s (77489 bytes in 0.003s)
/sdcard/screenshots/com.papershift.design.test/screenshots-default/screenshot_bundle.zip: 1 file pulled, 0 skipped. 173.7 MB/s (2248492 bytes in 0.012s)
/Users/runner/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.13.0/db0cdddf8ac23f29bb95a7aff50bb68142ce5fea/plugin-0.13.0.jar/android_screenshot_tests/pull_screenshots.py:154: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/runner/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.13.0/db0cdddf8ac23f29bb95a7aff50bb68142ce5fea/plugin-0.13.0.jar/android_screenshot_tests/pull_screenshots.py", line 652, in <module>
  File "/Users/runner/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.13.0/db0cdddf8ac23f29bb95a7aff50bb68142ce5fea/plugin-0.13.0.jar/android_screenshot_tests/pull_screenshots.py", line 648, in main
  File "/Users/runner/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.13.0/db0cdddf8ac23f29bb95a7aff50bb68142ce5fea/plugin-0.13.0.jar/android_screenshot_tests/pull_screenshots.py", line 570, in pull_screenshots
  File "/Users/runner/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.13.0/db0cdddf8ac23f29bb95a7aff50bb68142ce5fea/plugin-0.13.0.jar/android_screenshot_tests/recorder.py", line 131, in verify
android_screenshot_tests.recorder.VerifyError: Image screenshots/SearchablePicker_designSizeUncollapsedSelected.png is not same as /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpg9ADKg/SearchablePicker_designSizeUncollapsedSelected.png
Pulled 332 files from device

> Task :design:verifyDebugAndroidTestScreenshotTest FAILED

FAILURE: Build failed with an exception.

but when I run locally the tests are passing. NOTE: Every time when I generate the local images via "./gradlew recordDebugAndroidTestScreenshotTest" this command, the same image size is different everytime.

xiphirx commented 3 years ago

You'll have to figure out why the image is a different size, but will continue to fail unless the screenshot image sizes are stable.