search

pedrovgs / Shot

Screenshot testing library for Android
Apache License 2.0
1.19k stars 116 forks source link
android gradle-plugin java jetpack-compose screenshot-tests testing testing-tools

readme

Karumi logoShot Build, lint, and test

🔝 Top sponsors 🔝

EmergeLogo

Shot is an Android project you can use to write screenshot for your apps in a simple and friendly way.

What is this?

Shot is a Gradle plugin and a core android library thought to run screenshot tests for Android. This project provides a handy interface named ScreenshotTest and a ready to use ShotTestRunner you can use in order to write tests like these:

class GreetingScreenshotTest : ScreenshotTest {

    // If you are using regular Android views

    @Test
    fun theActivityIsShownProperly() {
        val mainActivity = startMainActivity()

        compareScreenshot(activity)
    }

    // If you are using Jetpack Compose

    @Test
    fun rendersGreetingMessageForTheSpecifiedPerson() {
        composeRule.setContent { Greeting(greeting) }

        compareScreenshot(composeRule)
    }
}

Since Shot 5.0.0 we provide screenshot testing support for Jetpack Compose. If you are testing your components using Shot we strongly recommend you to configure your emulator using the gpu mode swiftshader_indirect. This will help you to avoid rendering issues when verifying your screenshots from any CI environment.

smallVerificationReport1 smallVerificationReport2

Record your screenshots executing ./gradlew executeScreenshotTests -Precord

recording

And verify your tests executing ./gradlew executeScreenshotTests

verifying

If Shot finds any error in your tests execution the Gradle plugin will show a report as follows:

errorReport

You can find the complete Facebook SDK documentation here.

Getting started

Setup the Gradle plugin:

Modify your root build.gradle file:

  buildscript {
    // ...
    dependencies {
      // ...
      classpath 'com.karumi:shot:<LATEST_RELEASE>'
    }
  }

Apply the plugin from any of the modules configured in your project. In a simple Android app your app/build.gradle file:

apply plugin: 'shot'

You will also have to configure the instrumentation test runner in your module build.gradle file as follows:

android {
    // ...
    defaultConfig {
        // ...
        testInstrumentationRunner "com.karumi.shot.ShotTestRunner"
    }
    // ...

We created this test runner for you extending from the Android one.

This plugin sets up a few convenience commands you can list executing ./gradlew tasks and reviewing the Shot associated tasks:

shotTasksHelp

If you are using flavors the available Shot gradle tasks will be configured based on your flavors and build types configuration. You can find all the available shot tasks by executing ./gradlew tasks. For example, if your app has two flavors: green and blue the list of available Shot tasks will be:

executeScreenshotTests - Checks the user interface screenshot tests. If you execute this task using -Precord param the screenshot will be regenerated.
blueDebugDownloadScreenshots - Retrieves the screenshots stored into the Android device where the tests were executed for the build BlueDebug
blueDebugExecuteScreenshotTests - Records the user interface tests screenshots. If you execute this task using -Precord param the screenshot will be regenerated for the build BlueDebug
blueDebugRemoveScreenshotsBefore - Removes the screenshots recorded before the tests execution from the Android device where the tests were executed for the build BlueDebug
blueDebugRemoveScreenshotsAfter - Removes the screenshots recorded after the tests execution from the Android device where the tests were executed for the build BlueDebug
greenDebugDownloadScreenshots - Retrieves the screenshots stored into the Android device where the tests were executed for the build GreenDebug
greenDebugExecuteScreenshotTests - Records the user interface tests screenshots. If you execute this task using -Precord param the screenshot will be regenerated for the build GreenDebug
greenDebugRemoveScreenshotsBefore - Removes the screenshots recorded before the tests execution from the Android device where the tests were executed for the build GreenDebug
greenDebugRemoveScreenshotsAfter - Removes the screenshots recorded after the tests execution from the Android device where the tests were executed for the build GreenDebug

You can specify a directory suffix where screenshot & report will be saved ({FLAVOR}/{BUILD_TYPE}/{DIRECTORY_SUFFIX}) :
./gradlew executeScreenshotTests -PdirectorySuffix=Api26

If for some reason you are running your tests on a different machine and you want to skip the instrumentation tests execution and just compare the sources remember you can use the following shot configuration:

  shot {
    runInstrumentation = false
  }

Create this AndroidManifest.xml file inside your androidTest folder.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="<YOUR_APP_ID>.test"
    android:sharedUserId="<YOUR_APP_ID>.uid">

</manifest>

You'll have to add the same android:sharedUserId="<YOUR_APP_ID>.uid" configuration to your app/AndroidManfiest.xml file in order to let the testing APK write into the SDCard.. If you don't do this, you can end up facing a weird error with this message while running your tests:

java.lang.RuntimeException: Failed to create the directory /sdcard/screenshots/com.example.snapshottesting.test/screenshots-default for screenshots. Is your sdcard directory read-only?

If you want to use Shot to test your Android libraries code, you will have to configure the testApplicationId parameter as follows

android {
    // ...
    defaultConfig {
        // ...
        testApplicationId "<MY_TEST_APPLICATION_ID>"
    }
    // ...

If you are using AGP 7.X in your Android library you may need to configure the applicationId as part of the Shot extension configuration as follows:

shot {
  applicationId = "com.myapp"
}

AGP introduced a bug in 7.0.1-alpha4 we are not able to fix without this workaround for now. In the future, once AGP fixes the bug, we will remove this extension property.

Be careful and do not use the same id you are using for any other of the installed apps.

Now you are ready to use the ScreenshotTest interface from your tests:

class MyActivityTest: ScreenshotTest {
    @Test
    fun theActivityIsShownProperly() {
            val mainActivity = startMainActivity();
           /*
             * Take the actual screenshot. At the end of this call, the screenshot
             * is stored on the device and the gradle plugin takes care of
             * pulling it and displaying it to you in nice ways.
             */
            compareScreenshot(activity);
    }
}

Since Shot 5.0.0, if you are using Jetpack Compose your tests will look like this:

class GreetingScreenshotTest : ScreenshotTest {

    @Test
    fun rendersGreetingMessageForTheSpecifiedPerson() {
        composeRule.setContent { Greeting(greeting) }

        compareScreenshot(composeRule)
    }

This interface is full of useful methods you can use to take your screenshots with your activities, dialogs fragments, view holders or even custom views

You can find a complete example in this repository under the folder named shot-consumer or review this kata.***

Now you are ready to record and verify your screenshot tests!

Using shot on API 28+

If you want to use Shot on devices running API >= 28, you will have to enable access to non-SDK interfaces.
Execute the following commands:

For Android 9 (API level 28)

adb shell settings put global hidden_api_policy_pre_p_apps 1
adb shell settings put global hidden_api_policy_p_apps 1

For Android 10 (API level 29) or higher

adb shell settings put global hidden_api_policy 1

Recording tests

You can record your screenshot tests executing this command:

./gradlew <Flavor><BuildType>ExecuteScreenshotTests -Precord

or

./gradlew executeScreenshotTests -Precord

This will execute all your integration tests and it will pull all the generated screenshots into your repository so you can easily add them to the version control system.

Executing tests

Once you have a bunch of screenshot tests recorded you can easily verify if the behaviour of your app is the correct one executing this command:

./gradlew <Flavor><BuildType>ExecuteScreenshotTests

or

./gradlew executeScreenshotTests

After executing your screenshot tests using the Gradle task <Flavor><BuildType>ExecuteScreenshotTests a report with all your screenshots will be generated.

ScreenshotTest interface

ScreenshotTest interface has been designed to simplify the usage of the library. These are the features you can use:

Before taking the screenshot, Shot performs some tasks in order to stabilize the screenshot. You can find the detail about the tasks performed in ScreenshotTest#disableFlakyComponentsAndWaitForIdle:

You can find examples of the usage of this interface and every feature mentioned inside the examples named shot-consumer and shot-consumer-flavors.

In case you need to override the inner dependency Shot includes in your project during the plugin configuration you can do it as follows:

dependencies {
  ....
  shotDependencies "com.karumi:shot-android:ANY_VERSION_YOU_WANT_TO_USE"
  ....
}

Keep in mind you'll need to use a compatible version or Shot won't work as expected.

ActivityScenario support

ActivityTestRule has been deprecated and now the usage of ActivityScenario is recommended. However, Shot needs to be executed from the instrumentation thread to be able to extract all the test metadata needed to record and verify screenshots. That's why we've created an ActivityScenario extension method named waitForActivity. This extension is needed get the activity instance from the instrumentation thread instead of running Shot from the app target thread. Using this extension you can write tests like this:

class MainActivityTest: ScreenshotTest {
    @Test
    fun rendersTheDefaultActivityState() {
        val activity = ActivityScenario.launch(MainActivity::class.java)

        compareScreenshot(activity)
    }
}

I hope we can find a better solution for this issue in the future.

Executing tests from Android Studio

Shot is a Gradle plugin and it does not integrate with AS by default. After running your tests, Shot Gradle plugin will fetch the screenshots generated during tests' execution and use them to check if your tests are passing or not. You always can run your tests from command line as explained above. However, if you want to run your tests from AS you can create a configuration like this:

asConfig

Keep in mind the debugger may not work if use use this option. If you want to debug your tests you can run them from Android Studio as you'd do with any other instrumentation test and you may need to execute this command before running your test:

adb rm -rf /storage/emulated/0/Download/screenshots/*

Executing tests in multiple devices

If after some time writing screenshot tests your build takes too long to run our recommendation is to run your tests in multiple devices. Sharding your tests execution will split your test suite into different devices so your tests execution time will be reduced. This feature is not designed to test the UI across different platforms or screen resolutions, to do that we'd recommend you to configure the size of the screenshot taken by modifing the view height and width. To run your tests in multiple devices you can use Composer and the official Gradle Plugin they provide. Composer will take all your tests and will split the test suite execution between all the connected devices. Remember, if you are going to use more than one device all the devices should use the same Android OS and the same screen resolution and density! Keep also in mind composer needs Gradle 5.4.1 to be able to run your tests using multiple devices.

Once you've configured composer to run your tests you only need to update Shot to use the composer task as the instrumentation test task as follows:

shot {
  useComposer = true
}

If you are using orchestrator remember to enable it in composer configuration:

composer {
  // ...
  withOrchestrator true
  // ...
}

Tolerance

Shot provides a simple mechanism to be able to configure a threshold value when comparing recorded images with the new ones during the verification stage. You may need to use tolerance in your tests when testing compose components because the API Shot uses to record screenshots depending on the device where your tests are executed. There are other scenarios where you may need to configure a tolerance value, but these are not so common. If you want to configure it you can use this config in your build.gradle file.

shot {
    tolerance =  0.1 // 0,1% tolerance
}

If you configure tolerance = 100 it means the tolerance threshold will be 100% and all your tests will pass even if they should fail...so be careful when configuring this param.

Take into account the instrumentationTestTask could be different if you use different flavors or build types. Remember also you should use Shot > 3.0.0 because this feature was introduced in this release!

CI Reporting

Shot generates an HTML report you can review at the end of the recording or verification build. However, if you are running Shot in a CI environment which does not support saving the reporting files generated by the build you can verify your tests using this command ./gradlew debugExecuteScreenshotTests -PprintBase64. This will change how Shot show comparision errors displaying a command you can copy and paste on your local terminal for every screenshot test failed.

If you want to see only failing tests in output, please add showOnlyFailingTestsInReports = true as an option to your build.gradle.

Running only some tests

You can run a single test or test class, just add the android.testInstrumentationRunnerArguments.class parameter within your gradle call. This option works for both modes, verification and recording, just remember to add the -Precord if you want to do the latter.

Running all tests in a class:

./gradlew <Flavor><BuildType>executeScreenshotTests -Pandroid.testInstrumentationRunnerArguments.class=com.your.package.YourClassTest

Running a single test:

./gradlew <Flavor><BuildType>executeScreenshotTests -Pandroid.testInstrumentationRunnerArguments.class=com.your.package.YourClassTest#yourTest

Using shot on API 29+

Since Shot 5.8.0 we provide support for devices running API >= 29. There is no need to configure android:requestLegacyExternalStorage="true" or provide any special storage permission in your test `AndroidManifest.

Development documentation

Inside this repository you'll find two types of projects. The first one is all the code related to the Gradle Plugin. The second one is all the code related to the example projects we use as consumers of the Gradle Plugin and also as end to end tests. Review the following folders when developing a new feature for Shot or fixing a bug:

Plugin related folders:

Consumers and end to end tests folder:

When developing a feature or fixing a bug, plugin related folders are the place where most of the code will be. When writing end to end tests you should review the consumers' folders. Unit or integration tests should be added in the first group of folders using all the tooling already installed and configured for you.

When talking about the IDE we should use, our recommendation is simple. We should use IntelliJ Idea for the Gradle Plugin related code and Android Studio for the consumers. Keep in mind the consumers' configuration is not linked with the root Gradle file so you won't be able to build the consumers from the root of this project using Android Studio. That's why we recommend the usage of different IDEs using different root Gradle configurations when working with this repository.

Steps to develop a feature or fix a bug:

In case you need to start an Android emulator you have scripts inside the consumers' folder you can execute to create the emulators you'll need to run the tests :smiley:

This is the list of most useful Gradle tasks you might need divided by type of project:

iOS support

If you want to apply the same testing technique on iOS you can use Swift Snapshot Testing

Developed By

Follow me on Twitter

Add me to Linkedin

License

Copyright 2022 Pedro Vicente Gómez Sánchez

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.