Save test report to /build/reports/screenshotTests

[marcinkunert]

So far the report is being saved to a random(?) location file:///var/folders/pv/hgq81xls64s8lmbt8yk8_k780000gp/T/screenshotskwILGm/index.html. Is there a way to save it to build directory like e.g. /build/reports/screenshotTests? If not than have you considered adding such feature?

[dimorinny]

Yes, you should add referenceDir param to facebook screenshot configuration in your build.gradle. Like this:

screenshots {
    customTestRunner = true
    referenceDir = "$projectDir/pulled-screenshots" // <- here
    targetPackage = "com.avito.android.design.app"
}

This param will pass to pull screenshots script, and if this param is not specified, this script create temp directory: https://github.com/facebook/screenshot-tests-for-android/blob/master/plugin/src/py/android_screenshot_tests/pull_screenshots.py#L251

[marcinkunert]

Thanks for the tip. Unfortunately this param is only used when executing pullScreenshotsFromDirectory, not the usual screenshotTests (which uses pullScreenshots). I've tried to run with pullScreenshotsFromDirectory but it fails with error message:

>>> Using (/Users/marcin/Projects/Grafikowy/screenshot-tests/build/reports/screenshots) for screenshot verification
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/marcin/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.4.2/eb92d3dc1fe3d794de2cc0242aabda1e17fd634e/plugin-0.4.2.jar/android_screenshot_tests/pull_screenshots.py", line 330, in <module>
  File "/Users/marcin/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.4.2/eb92d3dc1fe3d794de2cc0242aabda1e17fd634e/plugin-0.4.2.jar/android_screenshot_tests/pull_screenshots.py", line 327, in main
  File "/Users/marcin/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.4.2/eb92d3dc1fe3d794de2cc0242aabda1e17fd634e/plugin-0.4.2.jar/android_screenshot_tests/pull_screenshots.py", line 258, in pull_screenshots
  File "/Users/marcin/.gradle/caches/modules-2/files-2.1/com.facebook.testing.screenshot/plugin/0.4.2/eb92d3dc1fe3d794de2cc0242aabda1e17fd634e/plugin-0.4.2.jar/android_screenshot_tests/pull_screenshots.py", line 235, in _validate_metadata
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 647, in parse
    source = open(source, "rb")
IOError: [Errno 2] No such file or directory: u'/Users/marcin/Projects/Grafikowy/screenshot-tests/build/reports/screenshots/metadata.xml'

Seems like a problem with pulling metadata.xml. I've checked and the file is available on the emulator: /storage/emulated/0/screenshots/com.example.android.testing.blueprint.test/screenshots-default/metadata.xml.

My current config:

screenshots {
//    customTestRunner = true
    referenceDir = "$projectDir/build/reports/screenshots"
    targetPackage = "com.example.android.testing.blueprint.test"
}

[dimorinny]

Oh, yes, you're right.

This param used only for pullScreenshotsFromDirectory. I think that you should create pull request with adding param for this case.

About pullScreenshotsFromDirectory:

Did you copy screenshots and metadata from your device manually? You have to copy all data (with metadata.xml) to /Users/marcin/Projects/Grafikowy/screenshot-tests/build/reports/screenshots/.

[marcinkunert]

I've tried that and now it results in another error. It tries to compare the screenshots with not existing samples. In my case I'm not really interested in comparing anything.

I'll probably end up creating a PR with the additional params for screenshotTests. The error messages could also be made a bit nicer.

[marcinkunert]

Done witch #87