search

takahirom / roborazzi

Make JVM Android integration test visible 🤖📸
https://takahirom.github.io/roborazzi/
Apache License 2.0
654 stars 25 forks source link

Unit test task with parameter ignore when using gradle.properties #276

Closed carlos-prezzee closed 3 months ago

carlos-prezzee commented 4 months ago

Hi 👋

I run into this issue lately

I have a gradle.properties file with

roborazzi.test.verify=true

But when running the gradle command with verify as false, the flag is ignore.

./gradlew testDebugUnitTest -Proborazzi.test.verify=false

Is it an expected scenario that gradle.properties will be the highest priority regardless of the gradle command?

takahirom commented 4 months ago

@carlos-prezzee Thank you! I'm not certain about the priorities of the flags, but I believe this behavior is not appropriate. Could you run ./gradlew testDebugUnitTest -Proborazzi.test.verify=false --info | grep Roborazzi to generate the log, and then add com.github.takahirom.roborazzi.ROBORAZZI_DEBUG = true to your test and share the logs with us?

carlos-prezzee commented 4 months ago

Here are the logs @takahirom Let me know if I can help in any other way

Roborazzi: Set output dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/test-results/roborazzi/results to test task
Roborazzi: Set output file /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/test-results/roborazzi/results-summary.json to test task
Roborazzi: Set output file /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/reports/roborazzi/index.html to test task
Roborazzi: Set output dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/test-results/roborazzi/results to test task
Roborazzi: Set output file /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/test-results/roborazzi/results-summary.json to test task
Roborazzi: Set output file /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/reports/roborazzi/index.html to test task
Tasks to be executed: [task ':fetchRobolectricDependencies', task ':pzlibs:barcode-generator:checkKotlinGradlePluginConfigurationErrors', task ':pzlibs:barcode-generator:preBuild', task ':pzlibs:barcode-generator:preDebugBuild', task ':pzlibs:barcode-generator:generateDebugResValues', task ':pzlibs:barcode-generator:generateDebugResources', task ':pzlibs:barcode-generator:packageDebugResources', task ':pzlibs:barcode-generator:parseDebugLocalResources', task ':pzlibs:barcode-generator:processDebugManifest', task ':pzlibs:barcode-generator:generateDebugRFile', task ':pzlibs:barcode-generator:compileDebugKotlin', task ':pzlibs:barcode-generator:javaPreCompileDebug', task ':pzlibs:barcode-generator:compileDebugJavaWithJavac', task ':pzlibs:barcode-generator:bundleLibRuntimeToJarDebug', task ':pzlibs:barcode-generator:bundleLibCompileToJarDebug', task ':pzlibs:barcode-generator:preDebugUnitTestBuild', task ':pzlibs:barcode-generator:writeDebugAarMetadata', task ':pzlibs:barcode-generator:checkDebugUnitTestAarMetadata', task ':pzlibs:barcode-generator:compileDebugLibraryResources', task ':pzlibs:barcode-generator:generateDebugUnitTestResValues', task ':pzlibs:barcode-generator:mapDebugUnitTestSourceSetPaths', task ':pzlibs:barcode-generator:generateDebugUnitTestResources', task ':pzlibs:barcode-generator:mergeDebugUnitTestResources', task ':pzlibs:barcode-generator:extractDeepLinksDebug', task ':pzlibs:barcode-generator:processDebugUnitTestManifest', task ':pzlibs:barcode-generator:processDebugUnitTestResources', task ':pzlibs:barcode-generator:compileDebugUnitTestKotlin', task ':pzlibs:barcode-generator:javaPreCompileDebugUnitTest', task ':pzlibs:barcode-generator:compileDebugUnitTestJavaWithJavac', task ':pzlibs:barcode-generator:mergeDebugShaders', task ':pzlibs:barcode-generator:compileDebugShaders', task ':pzlibs:barcode-generator:generateDebugAssets', task ':pzlibs:barcode-generator:mergeDebugAssets', task ':pzlibs:barcode-generator:generateDebugUnitTestAssets', task ':pzlibs:barcode-generator:packageDebugAssets', task ':pzlibs:barcode-generator:mergeDebugUnitTestAssets', task ':pzlibs:barcode-generator:packageDebugUnitTestForUnitTest', task ':pzlibs:barcode-generator:generateDebugUnitTestConfig', task ':pzlibs:barcode-generator:processDebugJavaRes', task ':pzlibs:barcode-generator:processDebugUnitTestJavaRes', task ':pzlibs:barcode-generator:restoreOutputDirRoborazzi', task ':pzlibs:barcode-generator:testDebugUnitTest', task ':pzlibs:barcode-generator:finalizeTestRoborazziDebug', task ':pzlibs:barcode-generator:preReleaseBuild', task ':pzlibs:barcode-generator:generateReleaseResValues', task ':pzlibs:barcode-generator:generateReleaseResources', task ':pzlibs:barcode-generator:packageReleaseResources', task ':pzlibs:barcode-generator:parseReleaseLocalResources', task ':pzlibs:barcode-generator:processReleaseManifest', task ':pzlibs:barcode-generator:generateReleaseRFile', task ':pzlibs:barcode-generator:compileReleaseKotlin', task ':pzlibs:barcode-generator:javaPreCompileRelease', task ':pzlibs:barcode-generator:compileReleaseJavaWithJavac', task ':pzlibs:barcode-generator:bundleLibRuntimeToJarRelease', task ':pzlibs:barcode-generator:bundleLibCompileToJarRelease', task ':pzlibs:barcode-generator:preReleaseUnitTestBuild', task ':pzlibs:barcode-generator:writeReleaseAarMetadata', task ':pzlibs:barcode-generator:checkReleaseUnitTestAarMetadata', task ':pzlibs:barcode-generator:compileReleaseLibraryResources', task ':pzlibs:barcode-generator:generateReleaseUnitTestResValues', task ':pzlibs:barcode-generator:mapReleaseUnitTestSourceSetPaths', task ':pzlibs:barcode-generator:generateReleaseUnitTestResources', task ':pzlibs:barcode-generator:mergeReleaseUnitTestResources', task ':pzlibs:barcode-generator:extractDeepLinksRelease', task ':pzlibs:barcode-generator:processReleaseUnitTestManifest', task ':pzlibs:barcode-generator:processReleaseUnitTestResources', task ':pzlibs:barcode-generator:compileReleaseUnitTestKotlin', task ':pzlibs:barcode-generator:javaPreCompileReleaseUnitTest', task ':pzlibs:barcode-generator:compileReleaseUnitTestJavaWithJavac', task ':pzlibs:barcode-generator:mergeReleaseShaders', task ':pzlibs:barcode-generator:compileReleaseShaders', task ':pzlibs:barcode-generator:generateReleaseAssets', task ':pzlibs:barcode-generator:mergeReleaseAssets', task ':pzlibs:barcode-generator:generateReleaseUnitTestAssets', task ':pzlibs:barcode-generator:packageReleaseAssets', task ':pzlibs:barcode-generator:mergeReleaseUnitTestAssets', task ':pzlibs:barcode-generator:packageReleaseUnitTestForUnitTest', task ':pzlibs:barcode-generator:generateReleaseUnitTestConfig', task ':pzlibs:barcode-generator:processReleaseJavaRes', task ':pzlibs:barcode-generator:processReleaseUnitTestJavaRes', task ':pzlibs:barcode-generator:testReleaseUnitTest', task ':pzlibs:barcode-generator:finalizeTestRoborazziRelease', task ':pzlibs:barcode-generator:test']
Analytics other plugin to proto: Unknown plugin type io.github.takahirom.roborazzi.RoborazziPlugin$Inject expected enum IO_GITHUB_TAKAHIROM_ROBORAZZI_ROBORAZZIPLUGIN
34            android:name="com.github.takahirom.roborazzi.RoborazziTransparentActivity"
35            android:theme="@style/RoborazziTransparentTheme" />
Resolve mutations for :pzlibs:barcode-generator:restoreOutputDirRoborazzi (Thread[included builds,5,main]) started.
:pzlibs:barcode-generator:restoreOutputDirRoborazzi (Thread[included builds,5,main]) started.
> Task :pzlibs:barcode-generator:restoreOutputDirRoborazzi SKIPPED
Skipping task ':pzlibs:barcode-generator:restoreOutputDirRoborazzi' as task onlyIf 'Task satisfies onlyIf spec' is false.
34            android:name="com.github.takahirom.roborazzi.RoborazziTransparentActivity"
35            android:theme="@style/RoborazziTransparentTheme" />
Roborazzi: Set output dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/intermediates/roborazzi to test task
Roborazzi: test.outputs.upToDateWhen !(doesRoborazziRun:false && inputDirEmpty:false)
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
    Roborazzi Debug: roborazziSystemPropertyTaskType():
    roborazziRecordFilePathStrategy(): com.github.takahirom.roborazzi.RoborazziRecordFilePathStrategy$RelativePathFromCurrentDirectory@7a944d1b
    RoborazziContext: RoborazziContext(
      ruleOverrideRoborazziOptions=null,
    Roborazzi Debug: roborazziSystemPropertyTaskType():
    roborazziRecordFilePathStrategy(): com.github.takahirom.roborazzi.RoborazziRecordFilePathStrategy$RelativePathFromCurrentDirectory@7a944d1b
    RoborazziContext: RoborazziContext(
      ruleOverrideRoborazziOptions=null,
    Roborazzi Debug: roborazziSystemPropertyTaskType():
    roborazziRecordFilePathStrategy(): com.github.takahirom.roborazzi.RoborazziRecordFilePathStrategy$RelativePathFromCurrentDirectory@7a944d1b
    RoborazziContext: RoborazziContext(
      ruleOverrideRoborazziOptions=null,
    Roborazzi Debug: roborazziSystemPropertyTaskType():
    roborazziRecordFilePathStrategy(): com.github.takahirom.roborazzi.RoborazziRecordFilePathStrategy$RelativePathFromCurrentDirectory@7a944d1b
    RoborazziContext: RoborazziContext(
      ruleOverrideRoborazziOptions=null,
    Roborazzi Debug: roborazziSystemPropertyTaskType():
    roborazziRecordFilePathStrategy(): com.github.takahirom.roborazzi.RoborazziRecordFilePathStrategy$RelativePathFromCurrentDirectory@7a944d1b
    RoborazziContext: RoborazziContext(
      ruleOverrideRoborazziOptions=null,
Roborazzi: test.doLast Copy files from /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/intermediates/roborazzi
Resolve mutations for :pzlibs:barcode-generator:finalizeTestRoborazziDebug (Thread[Execution worker Thread 4,5,main]) started.
:pzlibs:barcode-generator:finalizeTestRoborazziDebug (Thread[Execution worker Thread 4,5,main]) started.
> Task :pzlibs:barcode-generator:finalizeTestRoborazziDebug SKIPPED
Roborazzi: roborazziTestFinalizer.onlyIf doesRoborazziRun false
Skipping task ':pzlibs:barcode-generator:finalizeTestRoborazziDebug' as task onlyIf 'Task satisfies onlyIf spec' is false.
Roborazzi: Set output dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/build/intermediates/roborazzi to test task
Roborazzi: test.outputs.upToDateWhen !(doesRoborazziRun:false && inputDirEmpty:false)
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Roborazzi: Set input dir /Users/carlos/Development/GitHub/consumer-mobile/android/pzlibs/barcode-generator/src/test/screenshots to test task
Resolve mutations for :pzlibs:barcode-generator:finalizeTestRoborazziRelease (Thread[Execution worker Thread 6,5,main]) started.
:pzlibs:barcode-generator:finalizeTestRoborazziRelease (Thread[Execution worker Thread 6,5,main]) started.
> Task :pzlibs:barcode-generator:finalizeTestRoborazziRelease SKIPPED
Roborazzi: roborazziTestFinalizer.onlyIf doesRoborazziRun false
Skipping task ':pzlibs:barcode-generator:finalizeTestRoborazziRelease' as task onlyIf 'Task satisfies onlyIf spec' is false.

5 tests completed, 5 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':pzlibs:barcode-generator:testReleaseUnitTest'.
takahirom commented 4 months ago

The log does not contain Roborazzi: test.doFirst, which indicates that Roborazzi did not run as a task. https://github.com/takahirom/roborazzi/blob/main/include-build/roborazzi-gradle-plugin/src/main/java/io/github/takahirom/roborazzi/RoborazziPlugin.kt#L306C1-L306C51

Could you clarify what you mean by "the flag is ignored"? Additionally, I am interested in knowing whether you are using RoborazziTaskType in your RoborazziOptions, since RoborazziTaskType is regarded as the most potent option. Therefore, if you specify RoborazziTaskType in the test, other options will be overridden.

carlos-prezzee commented 4 months ago

Thanks @takahirom for looking into this.

btw I am running:

./gradlew test -Proborazzi.test.verify=false

instead of 

./gradlew testDebugUnitTest -Proborazzi.test

which made the tests also run in the release target and since I also faced this issue and by adding the debugImplementation("androidx.compose.ui:ui-test-manifest:$compose_version") it fails due to unable to resolve intent for androidx.activity.ComponentActivity

it might not be related at all to roborazzi but to the compose test rule unable to complete in the release target. Will try other alternatives.

takahirom commented 4 months ago

@carlos-prezzee I think you can try adding the testsupport module and adding the Activity to the testsupport main source sets as described in https://github.com/robolectric/robolectric/pull/4736 .

takahirom commented 3 months ago

If you have any issues yet, let me know.