"INVALID(NO_SIGNATURE)" on running Espresso UI Test

ryoyakawai commented 5 years ago

Hi

I am trying to run exactly same task that is currently runs on other CI service. However, it is not woking well to run espresso UI Test on Bitrise for now. There might exist inappropriate configuration on my recipe, but I would appreciate to look through my configuration.

Thanks!

App and Test Code

here

Configurations that are related to the issue

....
 primary:
    steps:
    - activate-ssh-key@4.0.3:
        run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}'
    - git-clone@4.0.14: {}
    - cache-pull@2.0.1: {}
    - script@1.1.5:
        title: Do anything with Script step
    - gradle-runner@1.9.0:
        inputs:
        - gradlew_path: "./gradlew"
        - gradle_task: assemble assembleDebugAndroidTest
        - gradle_file: build.gradle
    - install-missing-android-tools@2.3.5:
        inputs:
        - gradlew_path: "$PROJECT_LOCATION/gradlew"
    - virtual-device-testing-for-android@1.0.5:
        inputs:
        - app_package_id: com.example.uitestsample
        - test_devices: Pixel2,26,en,portrait
        - inst_test_package_id: com.example.uitestsample
        - inst_use_orchestrator: 'true'
        - inst_test_runner_class: AndroidJUnitRunner
        - download_test_results: 'true'
        - environment_variables: directories-to-pull /sdcard/uitest/
        - test_type: instrumentation
....

The error I got:

Waiting for test results
- Validating
Failed to get test status, error: Failed to get test status: INVALID(NO_SIGNATURE)
|                                                                              |
+---+---------------------------------------------------------------+----------+
| x | virtual-device-testing-for-android@1.0.5 (exit code: 1)       | 12 sec   |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...se-steplib/steps-virtual-device-testing-for-android/issues |
| Source: ...thub.com/bitrise-steplib/steps-virtual-device-testing-for-android |
+---+---------------------------------------------------------------+----------+

bitce commented 5 years ago

Hi @ryoyakawai!

Sorry to hear about the issue, can you please send us the build URL? Also, what do you see on the test results page?

ryoyakawai commented 5 years ago

Hi @bitce Thank you for the reply!

The URL is here >> https://app.bitrise.io/build/baa3f3f1dd4c7500.

I think you can see from the URL above, however error output is this.

+------------------------------------------------------------------------------+

| (6) virtual-device-testing-for-android@1.0.5                                 |
+------------------------------------------------------------------------------+
| id: virtual-device-testing-for-android                                       |
| version: 1.0.5                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: go                                                                  |
| time: 2019-07-19T05:07:43Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
INFO[05:07:43] Start installing (golang) with apt-get       
INFO[05:07:43]  * [OK] Step dependency (go) installed, available. 
Configs:
- ApkPath: /bitrise/deploy/app-release-unsigned.apk
- TestTimeout: 900
- DirectoriesToPull: 
- EnvironmentVariables: directories-to-pull /sdcard/uitest/
- TestDevices:
---
Model    API Level   Locale     Orientation   
Pixel2   26          portrait   en            
---
- AppPackageID: com.example.uitestsample
- TestType: instrumentation
- TestApkPath: /bitrise/deploy/app-debug-androidTest.apk
- InstTestPackageID: com.example.uitestsample
- InstTestRunnerClass: AndroidJUnitRunner
- InstTestTargets: 
- UseOrchestrator: true
Upload APKs
=> APKs uploaded
Start test
=> Test started
Waiting for test results
- Validating
Failed to get test status, error: Failed to get test status: INVALID(NO_SIGNATURE)
|                                                                              |
+---+---------------------------------------------------------------+----------+
| x | virtual-device-testing-for-android@1.0.5 (exit code: 1)       | 12 sec   |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...se-steplib/steps-virtual-device-testing-for-android/issues |
| Source: ...thub.com/bitrise-steplib/steps-virtual-device-testing-for-android |
+---+---------------------------------------------------------------+----------+

....

+------------------------------------------------------------------------------+
|                               bitrise summary                                |
+---+---------------------------------------------------------------+----------+
|   | title                                                         | time (s) |
+---+---------------------------------------------------------------+----------+
| - | activate-ssh-key@4.0.3                                        | 1.34 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | git-clone@4.0.14                                              | 5.05 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | cache-pull@2.0.1                                              | 9.77 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | Do anything with Script step                                  | 1.33 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | gradle-runner@1.9.0                                           | 228 sec  |
+---+---------------------------------------------------------------+----------+
| ✓ | install-missing-android-tools@2.3.5                           | 16 sec   |
+---+---------------------------------------------------------------+----------+
| Update available: 2.3.5 -> 2.3.7                                             |
+---+---------------------------------------------------------------+----------+
| x | virtual-device-testing-for-android@1.0.5 (exit code: 1)       | 12 sec   |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...se-steplib/steps-virtual-device-testing-for-android/issues |
| Source: ...thub.com/bitrise-steplib/steps-virtual-device-testing-for-android |
+---+---------------------------------------------------------------+----------+
| - | android-lint@0.9.6                                            | 0.85 sec |
+---+---------------------------------------------------------------+----------+
| - | android-unit-test@0.11.1                                      | 0.81 sec |
+---+---------------------------------------------------------------+----------+
| ✓ | deploy-to-bitrise-io@1.6.0                                    | 6.44 sec |
+---+---------------------------------------------------------------+----------+
| - | cache-push@2.2.0                                              | 0.82 sec |
+---+---------------------------------------------------------------+----------+
| Total runtime: 283 sec                                                       |
+------------------------------------------------------------------------------+

Thanks!

bitce commented 5 years ago

Hi @ryoyakawai!

Sorry for the delayed answer! The issue is a minor one in your setup. For instrumentation tests, you need to run an APK you've created against the test version of that very same APK.

So in this instance, this isn't working, because you are trying to test an app-release-unsigned.apk for which you'd need a file called app-release-unsigned-androidTest.apk. But instead you have one called app-debug-androidTest.apk which you could test with an APK called app-debug.apk I think you get the idea :)

Hope this helps, let me know if you have any questions!

ryoyakawai commented 5 years ago

Hi @bitce

Thanks! My bad. It works well now.

Let me share one configuration which is required to update to run Virtual Device Testing. That is configuration of Gradle Runner task and that is following: "Gradle task to run" must be "assembleDebug assembleDebugAndroidTest" to use variable $BITRISE_APK_PATH as [[BETA] Virtual Device Testing for Android] > [APK path]

So I am not sure whether I am saying right things or not, but I think to add "$BITRISE_DEBUG_APK_PATH" as environemtn variable is one of the way to avoid this error.

Anyway, thank you for helping me to solve my issue!!

bitrise-steplib / steps-virtual-device-testing-for-android