# Roborazzi
**Make JVM Android Integration Test Visible**
## Roborazzi now supports [Robolectric Native Graphics (RNG)](https://github.com/robolectric/robolectric/releases/tag/robolectric-4.10) and enables screenshot testing.📣
## Why Choose Roborazzi?
### Why is screenshot testing important?
Screenshot testing is key to validate your app's appearance and functionality. It efficiently
detects visual issues and tests the app as users would use it, making it easier to spot problems.
It's quicker than writing many assert statements, ensuring your app looks right and behaves
correctly.
### What are JVM tests and why test with JVM instead of on Android?
JVM tests, also known as local tests, are placed in the test/ directory and are run on a developer's
PC or CI environment. On the other hand, device tests, also known as Instrumentation tests, are
written in the androidTest/ directory and are run on real devices or emulators. Device testing can
result in frequent failures due to the device environment, leading to false negatives. These
failures are often hard to reproduce, making them tough to resolve.
### Paparazzi and Roborazzi: A Comparison
Paparazzi is a great tool for visualizing displays within the JVM. However, it's incompatible with
Robolectric, which also mocks the Android framework.
Roborazzi fills this gap. It integrates with Robolectric, allowing tests to run with Hilt and
interact with components. Essentially, Roborazzi enhances Paparazzi's capabilities, providing a more
efficient and reliable testing process by capturing screenshots with Robolectric.
**Leveraging Roborazzi in Test Architecture: An Example**
**Integrating Roborazzi into the Architecture: An Example from DroidKaigi 2023 App**
In the DroidKaigi 2023 app, Roborazzi was introduced from the early stages of development as part of the architectural design. This integration allowed the team to verify changes throughout the development process. The specific architectural decisions and how they were implemented can be found [README](https://github.com/DroidKaigi/conference-app-2023#screenshot-testing-with-robolectric-native-graphics-rng-and-roborazzi).
# Try it out
Available on Maven Central.
### Add Robolectric
This is an example of adding Robolectric to your project:
https://github.com/takahirom/roborazzi-usage-examples/compare/b697...5c12
This library is dependent on Robolectric. Please see below to add Robolectric.
https://robolectric.org/getting-started/
### Add Roborazzi
This is an example of adding Roborazzi to your project:
https://github.com/takahirom/roborazzi-usage-examples/commit/3a02
To take screenshots, please use Robolectric 4.10 alpha 1 or later and please
add `@GraphicsMode(GraphicsMode.Mode.NATIVE)` to your test class.
```kotlin
@GraphicsMode(GraphicsMode.Mode.NATIVE)
```
# Build setup
Roborazzi is available on maven central.
This plugin simply creates Gradle tasks record, verify, compare and passes the configuration to the
test.
**build.gradle.kts**
The comparison image, saved as `[original]_compare.png`, is shown below:
You can check the test report in `build/reports/roborazzi/index.html`
This
uses [JetNew from Compose Samples](https://github.com/android/compose-samples/tree/main/JetNews).
You can check the pull request introducing Roborazzi to the
compose-samples [here](https://github.com/takahirom/compose-samples/pull/1/files).
### Add dependencies
| Description | Dependencies |
|-----------------|--------------------------------------------------------------------------------------|
| Core functions | `testImplementation("io.github.takahirom.roborazzi:roborazzi:[version]")` |
| Jetpack Compose | `testImplementation("io.github.takahirom.roborazzi:roborazzi-compose:[version]")` |
| JUnit rules | `testImplementation("io.github.takahirom.roborazzi:roborazzi-junit-rule:[version]")` |
| plugins | buildscript |
| Define plugin in root build.gradle.kts ```kotlin plugins { ... id("io.github.takahirom.roborazzi") version "[version]" apply false } ``` Apply plugin in module build.gradle.kts ```kotlin plugins { ... id("io.github.takahirom.roborazzi") } ``` | root build.gradle.kts ```kotlin buildscript { dependencies { ... classpath("io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[version]") } } ``` module build.gradle.kts ```kotlin plugins { ... id("io.github.takahirom.roborazzi") } ``` |
build.gradle version
| plugins | buildscript |
| Define plugin in root build.gradle ```groovy plugins { ... id "io.github.takahirom.roborazzi" version "[version]" apply false } ``` Apply plugin in module build.gradle ```groovy plugins { ... id 'io.github.takahirom.roborazzi' } ``` | root build.gradle ```groovy buildscript { dependencies { ... classpath "io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[version]" } } ``` module build.gradle ```groovy apply plugin: "io.github.takahirom.roborazzi" ``` |
| Use Roborazzi task | Use default unit test task | Description |
| `./gradlew recordRoborazziDebug` | `./gradlew testDebugUnitTest` after adding `roborazzi.test.record=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.record=true` | Record a screenshot Default output directory is `build/outputs/roborazzi` You can check a report under `build/reports/roborazzi/index.html` |
| `./gradlew compareRoborazziDebug` | `./gradlew testDebugUnitTest` after adding `roborazzi.test.compare=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.compare=true` | Review changes made to an image. This action will compare the current image with the saved one, generating a comparison image labeled as `[original]_compare.png`. It also produces a JSON file containing the diff information, which can be found under `build/test-results/roborazzi`. |
| `./gradlew verifyRoborazziDebug` | `./gradlew testDebugUnitTest` after adding `roborazzi.test.verify=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.verify=true` | Validate changes made to an image. If there is any difference between the current image and the saved one, the test will fail. |
| `./gradlew verifyAndRecordRoborazziDebug` | `./gradlew testDebugUnitTest` after adding `roborazzi.test.verify=true` and `roborazzi.test.record=true` to your gradle.properties file. or `./gradlew testDebugUnitTest -Proborazzi.test.verify=true -Proborazzi.test.record=true` | This task will first verify the images and, if differences are detected, it will record a new baseline. |
| `./gradlew clearRoborazziDebug` | This is not a test task. | Note: This is an experimental task. This task will clear the saved images. This task also deletes the cached images. Please be careful when using this task. |
# How to use
### Take a screenshot manually
You can take a screenshot by calling captureRoboImage().
app/src/test/java/../ManualTest.kt
```kotlin
import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.test.core.app.ActivityScenario
import androidx.test.espresso.Espresso.onView
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.annotation.GraphicsMode
// All you need to do is use the captureRoboImage function in the test!
import com.github.takahirom.roborazzi.captureRoboImage
// Tips: You can use Robolectric while using AndroidJUnit4
@RunWith(AndroidJUnit4::class)
// Enable Robolectric Native Graphics (RNG)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ManualTest {
@get:Rule
val composeTestRule = createAndroidComposeRule()
@Test
fun captureRoboImageSample() {
// Tips: You can use Robolectric with Espresso API
// launch
ActivityScenario.launch(MainActivity::class.java)
// Capture screen
onView(ViewMatchers.isRoot())
// If you don't specify a screenshot file name, Roborazzi will automatically use the method name as the file name for you.
// The format of the file name will be as follows:
// build/outputs/roborazzi/com_..._ManualTest_captureRoboImageSample.png
.captureRoboImage()
// Capture Jetpack Compose Node
composeTestRule.onNodeWithTag("AddBoxButton")
.onParent()
.captureRoboImage("build/compose.png")
}
}
```
Roborazzi supports the following APIs.
### Device configuration
You can configure the device by using the `@Config` annotation and `RobolectricDeviceQualifiers`.
### Integrate to your GitHub Actions
It is easy to integrate Roborazzi to your GitHub Actions.
#### Add a job to store screenshots
```yaml
name: store screenshots
on:
push
env:
GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"
jobs:
test:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v3.9.0
with:
distribution: 'zulu'
java-version: 19
- name: Gradle cache
uses: gradle/gradle-build-action@v2
- name: test
run: |
# Create screenshots
./gradlew app:recordRoborazziDebug --stacktrace
# Upload screenshots to GitHub Actions Artifacts
- uses: actions/upload-artifact@v3
with:
name: screenshots
path: app/build/outputs/roborazzi
retention-days: 30
```
#### Add a job to verify screenshots
```yaml
name: verify test
on:
push
env:
GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"
jobs:
test:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v3.9.0
with:
distribution: 'zulu'
java-version: 19
- name: Gradle cache
uses: gradle/gradle-build-action@v2
# Download screenshots from main branch
- uses: dawidd6/action-download-artifact@v2
with:
name: screenshots
path: app/build/outputs/roborazzi
workflow: test.yaml
branch: main
- name: verify test
id: verify-test
run: |
# If there is a difference between the screenshots, the test will fail.
./gradlew app:verifyRoborazziDebug --stacktrace
- uses: actions/upload-artifact@v3
if: ${{ always() }}
with:
name: screenshot-diff
path: app/build/outputs/roborazzi
retention-days: 30
- uses: actions/upload-artifact@v3
if: ${{ always() }}
with:
name: screenshot-diff-reports
path: app/build/reports
retention-days: 30
- uses: actions/upload-artifact@v3
if: ${{ always() }}
with:
name: screenshot-diff-test-results
path: app/build/test-results
retention-days: 30
```
#### Advanced workflow Sample: Compare Snapshot Results on Pull Requests
For those who are looking for a more advanced example, we have prepared a sample repository that
demonstrates how to use Roborazzi to compare snapshot results on GitHub pull requests. This sample
showcases the integration of Roborazzi with GitHub Actions workflows, making it easy to visualize
and review the differences between snapshots directly in the pull request comments.
Check out
the [roborazzi-compare-on-github-comment-sample](https://github.com/takahirom/roborazzi-compare-on-github-comment-sample)
repository to see this powerful feature in action and learn how to implement it in your own
projects.
Example of the comment
## RoborazziRule (Optional)
RoborazziRule is a JUnit rule for Roborazzi.
RoborazziRule is **optional**. You can use `captureRoboImage()` without this rule.
RoborazziRule has two features.
1. Provide context such as `RoborazziOptions` and `outputDirectoryPath` etc for `captureRoboImage()`.
2. Capture screenshots for each test when specifying RoborazziRule.options.captureType.
For example, The following code generates an output file
named `**custom_outputDirectoryPath**/**custom_outputFileProvider**-com.github.takahirom.roborazzi.sample.RuleTestWithPath.captureRoboImage.png` :
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTestWithPath {
@get:Rule
val roborazziRule = RoborazziRule(
options = Options(
outputDirectoryPath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/custom_outputDirectoryPath",
outputFileProvider = { description, outputDirectory, fileExtension ->
File(
outputDirectory,
"custom_outputFileProvider-${description.testClass.name}.${description.methodName}.$fileExtension"
)
}
),
)
@Test
fun captureRoboImage() {
launch(MainActivity::class.java)
// The file will be saved using the rule's outputDirectoryPath and outputFileProvider
onView(isRoot()).captureRoboImage()
}
}
```
### Generate gif image
```kotlin
@Test
fun captureRoboGifSample() {
onView(ViewMatchers.isRoot())
.captureRoboGif("build/test.gif") {
// launch
ActivityScenario.launch(MainActivity::class.java)
// move to next page
onView(withId(R.id.button_first))
.perform(click())
// back
pressBack()
// move to next page
onView(withId(R.id.button_first))
.perform(click())
}
}
```
### Generate gif with test rule
> **Note**
> You **don't need to use RoborazziRule** if you're using captureRoboImage().
With the JUnit test rule, you do not need to name the gif image,
and if you prefer, you can output the gif image **only if the test fails**.
This test will output this file.
`build/outputs/roborazzi/com.github.takahirom.roborazzi.sample.RuleTestWithOnlyFail_captureRoboGifSampleFail.gif`
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTestWithOnlyFail {
@get:Rule
val roborazziRule = RoborazziRule(
captureRoot = onView(isRoot()),
options = Options(
onlyFail = true,
captureType = RoborazziRule.CaptureType.Gif,
)
)
@Test
fun captureRoboLastImageSampleFail() {
// launch
ActivityScenario.launch(MainActivity::class.java)
// move to next page
onView(withId(R.id.button_first))
.perform(click())
// should fail because the button does not exist
// Due to failure, the gif image will be saved in the outputs folder.
onView(withId(R.id.button_first))
.perform(click())
}
}
```
### Generate Jetpack Compose gif with test rule
Test target
```kotlin
@Composable
fun SampleComposableFunction() {
var count by remember { mutableStateOf(0) }
Column(
Modifier
.size(300.dp)
) {
Box(
Modifier
.testTag("AddBoxButton")
.size(50.dp)
.clickable {
count++
}
)
(0..count).forEach {
Box(
Modifier
.size(30.dp)
)
}
}
}
```
Test (Just add RoborazziRule)
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ComposeTest {
@get:Rule
val composeTestRule = createAndroidComposeRule()
@get:Rule
val roborazziRule = RoborazziRule(
composeRule = composeTestRule,
captureRoot = composeTestRule.onRoot(),
options = RoborazziRule.Options(
RoborazziRule.CaptureType.Gif()
)
)
@Test
fun composable() {
composeTestRule.setContent {
SampleComposableFunction()
}
(0 until 3).forEach { _ ->
composeTestRule
.onNodeWithTag("AddBoxButton")
.performClick()
}
}
}
```
### RoborazziRule options
You can use some RoborazziRule options
```kotlin
/**
* This rule is a JUnit rule for roborazzi.
* This rule is optional. You can use [captureRoboImage] without this rule.
*
* This rule have two features.
* 1. Provide context such as `RoborazziOptions` and `outputDirectoryPath` etc for [captureRoboImage].
* 2. Capture screenshots for each test when specifying RoborazziRule.options.captureType.
*/
class RoborazziRule private constructor(
private val captureRoot: CaptureRoot,
private val options: Options = Options()
) : TestWatcher() {
/**
* If you add this annotation to the test, the test will be ignored by
* roborazzi's CaptureType.LastImage, CaptureType.AllImage and CaptureType.Gif.
*/
annotation class Ignore
data class Options(
val captureType: CaptureType = CaptureType.None,
/**
* output directory path
*/
val outputDirectoryPath: String = provideRoborazziContext().outputDirectory,
val outputFileProvider: FileProvider = provideRoborazziContext().fileProvider
?: defaultFileProvider,
val roborazziOptions: RoborazziOptions = provideRoborazziContext().options,
)
sealed interface CaptureType {
/**
* Do not generate images. Just provide the image path to [captureRoboImage].
*/
object None : CaptureType
/**
* Generate last images for each test
*/
data class LastImage(
/**
* capture only when the test fail
*/
val onlyFail: Boolean = false,
) : CaptureType
/**
* Generate images for Each layout change like TestClass_method_0.png for each test
*/
data class AllImage(
/**
* capture only when the test fail
*/
val onlyFail: Boolean = false,
) : CaptureType
/**
* Generate gif images for each test
*/
data class Gif(
/**
* capture only when the test fail
*/
val onlyFail: Boolean = false,
) : CaptureType
}
```
#### Image comparator custom settings
When comparing images, you may encounter differences due to minor changes related to antialiasing. You can use the options below to avoid this.
```kotlin
@get:Rule
val roborazziRule = RoborazziRule(
options = RoborazziRule.Options(
roborazziOptions = RoborazziOptions(
compareOptions = RoborazziOptions.CompareOptions(
changeThreshold = 0.01, // For 1% accepted difference
imageComparator = SimpleImageComparator(
maxDistance = 0.007F, // 0.001F is default value from Differ
vShift = 2, // Increasing the shift can help resolve antialiasing issues
hShift = 2 // Increasing the shift can help resolve antialiasing issues
)
)
)
)
)
```
### Experimental WebP support and other image formats
You can set `roborazzi.record.image.extension` to `webp` in your `gradle.properties` file to generate WebP images.
```kotlin
roborazzi.record.image.extension=webp
```
WebP is a lossy image format by default, which can make managing image differences challenging. To address this, we provide a lossless WebP image comparison feature.
To enable WebP support, add `testImplementation("io.github.darkxanter:webp-imageio:0.3.3")` to your `build.gradle.kts` file.
```kotlin
onView(ViewMatchers.withId(R.id.textview_first))
.captureRoboImage(
roborazziOptions = RoborazziOptions(
recordOptions = RoborazziOptions.RecordOptions(
imageIoFormat = LosslessWebPImageIoFormat(),
),
)
)
```
You can also use other image formats by implementing your own `AwtImageWriter` and `AwtImageLoader`.
```kotlin
data class JvmImageIoFormat(
val awtImageWriter: AwtImageWriter,
val awtImageLoader: AwtImageLoader
) : ImageIoFormat
```
### Dump mode
If you are having trouble debugging your test, try Dump mode as follows.
### Accessibility Check
Roborazzi Accessibility Checks is a library that integrates accessibility checks into Roborazzi.
Please refer to [Accessibility Check](https://github.com/takahirom/roborazzi/blob/main/roborazzi-accessibility-check/README.md)
### Roborazzi options
Please check out [RoborazziOptions](https://github.com/takahirom/roborazzi/blob/main/include-build/roborazzi-core/src/commonJvmMain/kotlin/com/github/takahirom/roborazzi/RoborazziOptions.kt) for available Roborazzi options.
| Capture | Code |
| ✅ Jetpack Compose's onNode() | ```kotlin composeTestRule.onNodeWithTag("AddBoxButton") .captureRoboImage() ``` |
| ✅ Espresso's onView() | ```kotlin onView(ViewMatchers.isRoot()) .captureRoboImage() ``` ```kotlin onView(withId(R.id.button_first)) .captureRoboImage() ``` |
| ✅ View |
```kotlin
val view: View = composeTestRule.activity.findViewById |
| ✅ Jetpack Compose lambda | ```kotlin captureRoboImage() { Text("Hello Compose!") } ``` |
|
Experimental🧪 ✅ Captures the entire screen, including dialogs | ```kotlin captureScreenRoboImage() ``` |
| ✅ Bitmap | ```kotlin val bitmap: Bitmap = createBitmap(100, 100, Bitmap.Config.ARGB_8888) .apply { applyCanvas { drawColor(android.graphics.Color.YELLOW) } } bitmap.captureRoboImage() ``` |
| Configuration | Code |
| ✅ Predefined device configuration | You can change the device configuration by adding `@Config` to the class or method. ```kotlin @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config(qualifiers = RobolectricDeviceQualifiers.Pixel5) class RoborazziTest { ``` ```kotlin @Test @Config(qualifiers = RobolectricDeviceQualifiers.Pixel5) fun test() { ``` |
| ✅ Night mode | ```kotlin @Config(qualifiers = "+night") ``` |
| ✅ Locale | ```kotlin @Config(qualifiers = "+ja") ``` |
| ✅ Screen size | ```kotlin @Config(qualifiers = RobolectricDeviceQualifiers.MediumTablet) ``` |
## RoborazziRule (Optional)
RoborazziRule is a JUnit rule for Roborazzi.
RoborazziRule is **optional**. You can use `captureRoboImage()` without this rule.
RoborazziRule has two features.
1. Provide context such as `RoborazziOptions` and `outputDirectoryPath` etc for `captureRoboImage()`.
2. Capture screenshots for each test when specifying RoborazziRule.options.captureType.
For example, The following code generates an output file
named `**custom_outputDirectoryPath**/**custom_outputFileProvider**-com.github.takahirom.roborazzi.sample.RuleTestWithPath.captureRoboImage.png` :
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTestWithPath {
@get:Rule
val roborazziRule = RoborazziRule(
options = Options(
outputDirectoryPath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/custom_outputDirectoryPath",
outputFileProvider = { description, outputDirectory, fileExtension ->
File(
outputDirectory,
"custom_outputFileProvider-${description.testClass.name}.${description.methodName}.$fileExtension"
)
}
),
)
@Test
fun captureRoboImage() {
launch(MainActivity::class.java)
// The file will be saved using the rule's outputDirectoryPath and outputFileProvider
onView(isRoot()).captureRoboImage()
}
}
```
### Generate gif image
```kotlin
@Test
fun captureRoboGifSample() {
onView(ViewMatchers.isRoot())
.captureRoboGif("build/test.gif") {
// launch
ActivityScenario.launch(MainActivity::class.java)
// move to next page
onView(withId(R.id.button_first))
.perform(click())
// back
pressBack()
// move to next page
onView(withId(R.id.button_first))
.perform(click())
}
}
```
### Generate gif with test rule
> **Note**
> You **don't need to use RoborazziRule** if you're using captureRoboImage().
With the JUnit test rule, you do not need to name the gif image,
and if you prefer, you can output the gif image **only if the test fails**.
This test will output this file.
`build/outputs/roborazzi/com.github.takahirom.roborazzi.sample.RuleTestWithOnlyFail_captureRoboGifSampleFail.gif`
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTestWithOnlyFail {
@get:Rule
val roborazziRule = RoborazziRule(
captureRoot = onView(isRoot()),
options = Options(
onlyFail = true,
captureType = RoborazziRule.CaptureType.Gif,
)
)
@Test
fun captureRoboLastImageSampleFail() {
// launch
ActivityScenario.launch(MainActivity::class.java)
// move to next page
onView(withId(R.id.button_first))
.perform(click())
// should fail because the button does not exist
// Due to failure, the gif image will be saved in the outputs folder.
onView(withId(R.id.button_first))
.perform(click())
}
}
```
### Generate Jetpack Compose gif with test rule
Test target
```kotlin
@Composable
fun SampleComposableFunction() {
var count by remember { mutableStateOf(0) }
Column(
Modifier
.size(300.dp)
) {
Box(
Modifier
.testTag("AddBoxButton")
.size(50.dp)
.clickable {
count++
}
)
(0..count).forEach {
Box(
Modifier
.size(30.dp)
)
}
}
}
```
Test (Just add RoborazziRule)
```kotlin
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ComposeTest {
@get:Rule
val composeTestRule = createAndroidComposeRule
# Experimental Compose Preview Support
Roborazzi provides support for generating screenshot tests and easy setup for Jetpack Compose Preview.
This support uses [ComposablePreviewScanner](https://github.com/sergio-sastre/ComposablePreviewScanner) to scan the Composable Previews in your project.
## Generate Compose Preview screenshot tests
You first need to add the Roborazzi plugin to your project. Please refer to the [setup guide](https://takahirom.github.io/roborazzi/build-setup.html) for more information.
Then you can enable the Compose Preview screenshot test generation feature by adding the following configuration to your `build.gradle.kts` file:
```kotlin
roborazzi {
generateComposePreviewRobolectricTests {
enable = true
}
}
```
The plugin will not automatically change your settings or add dependencies to prevent conflicts with your existing setup. However, it will provide instructions on what to do next, such as adding dependencies and required code. You can also check the [sample project](https://github.com/takahirom/roborazzi/tree/main/sample-generate-preview-tests) for a complete example.
After that, you can run the `recordRoborazziDebug` task to generate screenshots using the generated tests, as described in the [setup guide](https://takahirom.github.io/roborazzi/build-setup.html).
### Customizing the Preview screenshot test
You can customize the generated test by adding the following configuration to your `build.gradle.kts` file:
```kotlin
roborazzi {
@OptIn(ExperimentalRoborazziApi::class)
generateComposePreviewRobolectricTests {
enable = true
// The package names to scan for Composable Previews.
packages = listOf("com.example")
// robolectricConfig will be passed to Robolectric's @Config annotation in the generated test class.
// See https://robolectric.org/configuring/ for more information.
robolectricConfig = mapOf(
"sdk" to "[32]",
"qualifiers" to "RobolectricDeviceQualifiers.Pixel5",
)
// If true, the private previews will be included in the test.
includePrivatePreviews = true
// The fully qualified class name of the custom test class that implements [com.github.takahirom.roborazzi.ComposePreviewTester].
testerQualifiedClassName = "com.example.MyCustomComposePreviewTester"
}
}
```
> **Note**
> If you are using build.gradle instead of build.gradle.kts, you need to use the set method for each assignment, like
> ```kotlin
> generateComposePreviewRobolectricTests.packages.set(["com.example"])
> ```
## Manually adding Compose Preview screenshot tests
Roborazzi provides a helper function for ComposablePreviewScanner.
You can add the following dependency to your project to use the helper function:
`testImplementation("io.github.takahirom.roborazzi:roborazzi-compose-preview-scanner-support:[version]")`
Then you can use the `ComposablePreview.captureRoboImage()` function to capture the Composable Preview using the settings in Preview annotations.
To obtain the `ComposablePreview` object, please refer to [ComposablePreviewScanner](https://github.com/sergio-sastre/ComposablePreviewScanner).
```kotlin
fun ComposablePreview.captureRoboImage(
filePath: String,
roborazziOptions: RoborazziOptions
)
```
### The supported `@Preview` annotation options
Currently, we don't support all the annotation options provided by the Compose Preview.
You can check the supported annotations in the [source code](https://github.com/takahirom/roborazzi/blob/main/roborazzi-compose-preview-scanner-support/src/main/java/com/github/takahirom/roborazzi/RobolectricPreviewInfosApplier.kt).
We are looking forward to your contributions to support more annotation options.
# Experimental AI-Powered Image Assertion
Roborazzi supports AI-powered image assertion.
AI-powered image assertion is an experimental feature. Screenshot tests are a great way to verify your app's UI, but verifying the content of the images can be a tedious and time-consuming task. This manual effort reduces scalability. Roborazzi can help automate this process through AI-powered image assertion, making it more efficient and scalable.
There are two new library modules: `io.github.takahirom.roborazzi:roborazzi-ai-gemini` and `io.github.takahirom.roborazzi:roborazzi-ai-openai` for AI-powered image assertion.
`roborazzi-ai-gemini` leverages [Gemini](https://gemini.google.com/) and [generative-ai-kmp](https://github.com/PatilShreyas/generative-ai-kmp), while `roborazzi-ai-openai` utilizes the [OpenAI API](https://platform.openai.com/) through raw HTTP API calls implemented with Ktor and KotlinX Serialization
```kotlin
...
@get:Rule
val composeTestRule = createAndroidComposeRule()
@get:Rule
val roborazziRule = RoborazziRule(
options = RoborazziRule.Options(
roborazziOptions = RoborazziOptions(
compareOptions = RoborazziOptions.CompareOptions(
aiAssertionOptions = AiAssertionOptions(
aiAssertionModel = GeminiAiAssertionModel(
// DO NOT HARDCODE your API key in your code.
// This is an example passing API Key through unitTests.all{ environment(key, value) }
apiKey = System.getenv("gemini_api_key") ?: ""
),
)
)
)
)
)
@Test
fun captureWithAi() {
ROBORAZZI_DEBUG = true
onView(ViewMatchers.isRoot())
.captureRoboImage(
roborazziOptions = provideRoborazziContext().options.addedAiAssertions(
AiAssertionOptions.AiAssertion(
assertionPrompt = "it should have PREVIOUS button",
requiredFulfillmentPercent = 90,
),
AiAssertionOptions.AiAssertion(
assertionPrompt = "it should show First Fragment",
requiredFulfillmentPercent = 90,
)
)
)
}
```
## Behavior of AI-Powered Image Assertion
AI-Powered Image Assertion runs only when the images are different. If the images are the same, AI-Powered Image Assertion is skipped.
This is because AI-Powered Image Assertion can be slow and expensive.
## Manual Image Assertion
You can use manual image assertion with Roborazzi. This allows you to utilize local LLMs or other LLMs. Manual Image Assertion doesn't require adding any dependencies other than Roborazzi itself.
You must provide the `AiAssertionModel` to `RoborazziOptions` to use manual image assertion.
```kotlin
interface AiAssertionModel {
fun assert(
referenceImageFilePath: String,
comparisonImageFilePath: String,
actualImageFilePath: String,
aiAssertionOptions: AiAssertionOptions
): AiAssertionResults
}
```
```kotlin
compareOptions = RoborazziOptions.CompareOptions(
aiAssertionOptions = AiAssertionOptions(
aiAssertionModel = object : AiAssertionOptions.AiAssertionModel {
override fun assert(
comparisonImageFilePath: String,
aiAssertionOptions: AiAssertionOptions
): AiAssertionResults {
// You can use any LLMs here to create AiAssertionResults
return AiAssertionResults(
aiAssertionResults = aiAssertionOptions.aiAssertions.map { assertion ->
AiAssertionResult(
assertionPrompt = assertion.assertionPrompt,
fulfillmentPercent = fulfillmentPercent,
requiredFulfillmentPercent = assertion.requiredFulfillmentPercent,
failIfNotFulfilled = assertion.failIfNotFulfilled,
explanation = "This is a manual test.",
)
}
)
}
},
aiAssertions = listOf(
AiAssertionOptions.AiAssertion(
assertionPrompt = "it should have PREVIOUS button",
requiredFulfillmentPercent = 90,
),
),
)
)
...
```
# Roborazzi IntelliJ IDEA and Android Studio Plugin
[https://plugins.jetbrains.com/plugin/24561-roborazzi](https://plugins.jetbrains.com/plugin/24561-roborazzi)
Demo of Roborazzi IntelliJ IDEA and Android Studio Plugin.
[https://github.com/takahirom/roborazzi/assets/1386930/bd3e8106-1971-4ec4-b764-3f1831999228](https://github.com/takahirom/roborazzi/assets/1386930/bd3e8106-1971-4ec4-b764-3f1831999228)
The Roborazzi plugin enhances your development workflow by automatically displaying screenshots related to Roborazzi's screenshot tests within IntelliJ IDEA. Easily view and verify visual outputs directly in your IDE, streamlining your testing and development process.
# Multiplatform support
### Experimental feature: iOS support
Roborazzi supports Compose Multiplatform iOS. You can use Roborazzi with Compose iOS as follows:
Gradle settings
```kotlin
plugins {
kotlin("multiplatform")
id("org.jetbrains.compose")
id("io.github.takahirom.roborazzi")
}
kotlin {
sourceSets {
...
val appleTest by getting {
dependencies {
implementation(project("io.github.takahirom.roborazzi:roborazzi-compose-ios:[1.13.0 or higher]"))
implementation(kotlin("test"))
}
}
...
```
Test with Roborazzi
```kotlin
class IosTest {
@OptIn(ExperimentalTestApi::class)
@Test
fun test() {
runComposeUiTest {
setContent {
MaterialTheme {
Column {
Button(
modifier = Modifier.alpha(alpha),
onClick = { }) {
Text("Hello World")
}
Box(
modifier = Modifier
.background(Color.Red.copy(alpha = alpha), MaterialTheme.shapes.small)
.size(100.dp),
)
}
}
}
onRoot().captureRoboImage(this, filePath = "ios.png")
onNodeWithText("Hello World").captureRoboImage(
this,
filePath = "ios_button.png"
)
}
}
}
```
Then, you can run the Gradle tasks for iOS Support, just like you do for Android Support.
```
./gradlew recordRoborazzi[SourceSet]
```
```
./gradlew recordRoborazziIosSimulatorArm64
./gradlew compareRoborazziIosSimulatorArm64
./gradlew verifyRoborazziIosSimulatorArm64
...
```
The currently implemented features for iOS support are as follows:
| Feature | status |
|---|---|
| Record | supported |
| Compare | supported |
| Verify | supported |
| Report | supported |
| Dropbox/Differ comparison | 🆖 |
| dump | 🆖 |
| resizing image | 🆖 |
| context data | 🆖 |
| custom reporter | 🆖 |
| RoborazziRecordFilePathStrategy | 🆖 |
| ComparisonStyle | 🆖 |
| resultValidator | 🆖 |
| resultValidator | 🆖 |
| applyDeviceCrop | 🆖 |
| pixelBitConfig | 🆖 |
We are migrating JVM implementation to Multiplatform implementation. So, some features are not supported yet.
We are looking for contributors to help us implement these features.
### Experimental feature: Compose Desktop support
Roborazzi supports Compose Desktop. You can use Roborazzi with Compose Desktop as follows:
Gradle settings
```kotlin
plugins {
kotlin("multiplatform")
id("org.jetbrains.compose")
id("io.github.takahirom.roborazzi")
}
kotlin {
// You can use your source set name
jvm("desktop")
sourceSets {
...
val desktopTest by getting {
dependencies {
implementation(project("io.github.takahirom.roborazzi:roborazzi-compose-desktop:[1.6.0-alpha-2 or higher]"))
implementation(kotlin("test"))
}
}
...
```
Test target Composable function
```kotlin
@Composable
fun App() {
var text by remember { mutableStateOf("Hello, World!") }
MaterialTheme {
Button(
modifier = Modifier.testTag("button"),
onClick = {
text = "Hello, Desktop!"
}) {
Text(
style = MaterialTheme.typography.h2,
text = text
)
}
}
}
```
Test with Roborazzi
```kotlin
class MainKmpTest {
@OptIn(ExperimentalTestApi::class)
@Test
fun test() = runDesktopComposeUiTest {
setContent {
App()
}
val roborazziOptions = RoborazziOptions(
recordOptions = RoborazziOptions.RecordOptions(
resizeScale = 0.5
),
compareOptions = RoborazziOptions.CompareOptions(
changeThreshold = 0F
)
)
onRoot().captureRoboImage(roborazziOptions = roborazziOptions)
onNodeWithTag("button").performClick()
onRoot().captureRoboImage(roborazziOptions = roborazziOptions)
}
}
```
Then, you can run the Gradle tasks for Desktop Support, just like you do for Android Support.
```
./gradlew recordRoborazzi[SourceSet]
```
```
./gradlew recordRoborazziDesktop
./gradlew compareRoborazziDesktop
./gradlew verifyRoborazziDesktop
...
```
If you use the Kotlin JVM plugin, the task will be `recordRoborazzi**Jvm**`.
The sample image
# Roborazzi gradle.properties Options and Recommendations
You can configure the following options in your `gradle.properties` file.
You can also use `-P` to set the options in the command line. For example, `./gradlew test -Proborazzi.test.record=true`.
## roborazzi.test
This option enables you to configure the behavior of Roborazzi. By default, all settings are set to false.
For additional configuration options, please refer to the 'Apply Roborazzi Gradle Plugin' section.
```
roborazzi.test.record=true
# roborazzi.test.compare=true
# roborazzi.test.verify=true
```
## roborazzi.record
### roborazzi.record.resizeScale
This option lets you set the resize scale for the image being recorded. The default value is 1.0.
```
roborazzi.record.resizeScale=0.5
```
### roborazzi.record.filePathStrategy
This setting allows you to specify the file path strategy for the recorded image. The default strategy is `relativePathFromCurrentDirectory`. If you choose `relativePathFromRoborazziContextOutputDirectory`, the file will be saved in the output directory specified by `RoborazziRule.Options.outputDirectoryPath`.
```
roborazzi.record.filePathStrategy=relativePathFromRoborazziContextOutputDirectory
```
### roborazzi.record.namingStrategy
This option enables you to define the naming strategy for the recorded image. The default strategy is `testPackageAndClassAndMethod`.
- If you choose `testPackageAndClassAndMethod`, the file name will be `com.example.MyTest.testMethod.png`.
- If you choose `escapedTestPackageAndClassAndMethod`, the file name will be `com_example_MyTest.testMethod.png`.
- If you choose `testClassAndMethod`, the file name will be `MyTest.testMethod.png`.
```
roborazzi.record.namingStrategy=testClassAndMethod
```
## roborazzi.cleanupOldScreenshots
This option allows you to clean up old screenshots. By default, this option is set to false.
The reason why Roborazzi does not delete old screenshots by default is that Roborazzi doesn't know if you are running filtered tests or not. If you are running filtered tests, Roborazzi will delete the screenshots that are not related to the current test run.
```
roborazzi.cleanupOldScreenshots=true
```
## Robolectric Options
### robolectric.pixelCopyRenderMode
I recommend setting `it.systemProperties["robolectric.pixelCopyRenderMode"] = "hardware"` in your `build.gradle` file to enhance the accuracy of your screenshots. For more details, please refer to the [issue](https://github.com/takahirom/roborazzi/issues/296).
```
android {
testOptions {
...
unitTests {
isIncludeAndroidResources = true
isReturnDefaultValues = true
all {
it.systemProperties["robolectric.pixelCopyRenderMode"] = "hardware"
}
```
# FAQ
### Q: How can I run only screenshot tests using Roborazzi?
**A:** To run only screenshot tests, you can configure your project with the following:
```groovy
android {
testOptions {
unitTests {
all {
// -Pscreenshot to filter screenshot tests
it.useJUnit {
if (project.hasProperty("screenshot")) {
includeCategories("io.github.takahirom.roborazzi.testing.category.ScreenshotTests")
}
}
}
}
}
}
```
Include the `-Pscreenshot` property, and only the screenshot tests will be run.
Note: This feature is not provided in the Roborazzi library itself, to keep it simple and utilize JUnit's built-in features for test filtering.
You can also annotate your tests like this:
```kotlin
/**
* You can filter ScreenshotTests using -Pscreenshot parameter
*/
interface ScreenshotTests
@Test
@Category(ScreenshotTests::class)
fun checkLaunchShot() {
onRoot().captureRoboImage(roborazziOptions = roborazziOptions)
}
```
This allows you to create a category of screenshot tests and filter them using the `-Pscreenshot` property, thus making it easier to run only those specific tests.
### Q: How can I debug screenshot tests in Android Studio?
### Q: How can I execute screenshot tests using Android Studio's Run button?
A: To execute screenshot tests using Android Studio's Run button, configure your project as follows:
`gradle.properties`
```groovy
roborazzi.test.record=true
```
After that, you can execute screenshot tests using either Android Studio's Run or Debug button as you normally would.
### Q: My screenshot tests are not capturing images. What could be the issue?
**A:** If your screenshot tests are not capturing images, there may be several patterns that are causing this issue. Please follow these troubleshooting steps:
- **Enable Debugging**: Set `ROBORAZZI_DEBUG = true` to see logs.
- **Check Plugin**: Ensure that the plugin is properly applied.
- **Run Task**: Verify that the `recordRoborazziDebug` task is running.
- **Call Method**: Confirm that `captureRoboImage()` is being called.
By following these steps, you should be able to identify and resolve the issue causing the screenshot tests to not capture images.
### Q: I'm seeing an optimization warning related to Java lambdas in Gradle. What can I do?
**A:** This warning may occur with Gradle 7.5. Upgrade to Gradle 7.6.2 to resolve this issue. Change the distribution URL in `gradle-wrapper.properties`:
```
distributionUrl=https\://services.gradle.org/distributions/gradle-7.6.2-bin.zip
```
### Q: Can I run Roborazzi with Bazel?
**A:** As of now, there is no direct support for running Roborazzi with Bazel. However, it is possible to do so. Please refer to the following comment for more details:
[Roborazzi Bazel Support Comment](https://github.com/takahirom/roborazzi/issues/63#issuecomment-1531990825)
### Q: My tests are being skipped or, conversely, are being run when they should be skipped. How can I handle caching to address this?
**A:** The behavior you are experiencing may be related to caching issues. Although it's
experimental, you can set the `outputDir` parameter in your `build.gradle` file to handle caching
and improve the stability of your tests. This parameter allows you to specify the output directory
for your screenshots, which can help in managing the cache. Here is how you can set it up:
If you use the default output directory(module/build/outputs/roborazzi), specifying the `outputDir`
parameter is not necessary. For more reference, you can check
out [the issue](https://github.com/takahirom/roborazzi/issues/193#issuecomment-1782073746).
```gradle
roborazzi {
outputDir = "src/your/screenshot/folder"
}
```
### Q: Why do my screenshot tests fail inconsistently across different operating systems like MacOS, Ubuntu, and Windows?
**A:** This is a known issue caused by variations in how graphics libraries render components on different platforms.
This issue is due to variations in how graphics libraries render components on different platforms.
As we've seen with [Now in Android](https://github.com/android/nowinandroid/issues/1242#issuecomment-2032962982), there are no guarantees for identical rendering across all environments.
To ensure more consistent results, consider configuring your continuous integration (CI) environment to both record and test screenshots. This setup minimizes discrepancies by using the same settings and resources for all tests.
### Q: The images taken from Roborazzi seem broken
**A:** First, please check if you are using the correct version of the Android SDK. Consider using API level 28 or higher because it seems that Robolectric Native Graphics doesn't work property on older SDK versions.
You can specify the SDK version like `@Config(sdk = [33])` or by using `robolectric.properties` with `sdk=33`.
[https://robolectric.org/configuring/](https://robolectric.org/configuring/)
If your images are mostly fine but there are some drawing issues like shadows or shape graphics, you can try using Robolectric's Hardware Rendering mode. Please refer to [this issue](https://github.com/takahirom/roborazzi/issues/255#issuecomment-1972838571).
### Q: I am seeing Out Of Memory errors.
**A:** You may solve this by using `unitTests.maxHeapSize` to adjust the unit test heap size as follows:
```groovy
android {
...
testOptions {
unitTests.all {
maxHeapSize = "4096m"
}
}
}
```
It is discussed in [this issue](https://github.com/takahirom/roborazzi/issues/272).
Additionally, it might be worth trying to run your tests with VisualVM to monitor memory usage and identify potential leaks.
### Q: [IDEA Plugin] Roborazzi Gradle task is not displayed in Tool Window.
**A:** It is discussed in [this issue](https://github.com/takahirom/roborazzi/issues/493).
To enable the display of Roborazzi tasks, please enable ***Configure all Gradle tasks during Gradle Sync (this can make Gradle Sync slower)*** in the Settings | Experimental | Gradle.
LICENSE
Copyright 2023 takahirom
Copyright 2019 Square, Inc.
Copyright The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.