search

android / android-test

An extensive framework for testing Android apps
https://android.github.io/android-test
Apache License 2.0
1.16k stars 314 forks source link

Instrumentation did not complete failure when no tests are scheduled to run #2265

Open kartik1o opened 2 months ago

kartik1o commented 2 months ago

Description

I have stumbled upon an issue where, if my instrumentation test filter is not able to run any tests, gradle build fails. Is there a way to avoid this ?

I have a test suite consisting of tablet and phone tests, and during runtime I am doing custom filtering based on the characteristic of device under test using CustomFilter API's. So, I have device pool consisting of a phone and tablet device. And I have having a class level annotation that specify tests to only target the tablet devices. As expected, all the tests run on tablet device. Since, there were no tests scheduled for the phone device. Even after all the test finished successfully, the build failed with the following error: 

> Task :app:connectedDebugAndroidTest
Starting 0 tests on Pixel_6_API_34(AVD) - 14

Finished 0 tests on Pixel_6_API_34(AVD) - 14

> Task :app:connectedDebugAndroidTest
Starting 4 tests on Pixel_Tablet_API_34(AVD) - 14

Pixel_Tablet_API_34(AVD) - 14 Tests 1/4 completed. (0 skipped) (0 failed)
Pixel_Tablet_API_34(AVD) - 14 Tests 2/4 completed. (0 skipped) (0 failed)
Pixel_Tablet_API_34(AVD) - 14 Tests 3/4 completed. (0 skipped) (0 failed)
Finished 4 tests on Pixel_Tablet_API_34(AVD) - 14

> Task :app:connectedDebugAndroidTest
Instrumentation did not complete:
Instrumentation code: 0
Instrumentation result bundle: 

> Task :app:connectedDebugAndroidTest FAILED
INSTRUMENTATION_RESULT: stream=

Time: 0

OK (0 tests)

INSTRUMENTATION_CODE: 0

this is the test-results.log file generated under app/build/outputs/androidTest-results dir for Pixel_6 device. I suspect instrumentation code need to be -1 for instrumentation to succeed. 

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite tests="0" failures="0" errors="0" skipped="0" time="0.61" timestamp="2024-08-12T14:46:15" hostname="localhost">
  <properties>
    <property name="device" value="Pixel_6_API_34(AVD) - 14" />
    <property name="project" value=":app" />
  </properties>
  <system-err>Instrumentation did not complete:
Instrumentation code: 0
Instrumentation result bundle: 
</system-err>
</testsuite>

xml report seems to contain system-err tag generated for the Pixel_6 device.

Steps to Reproduce

Spin up an emultor with tablet resolution(landscape) and other one with phone resolution(portrait) Run test as usually and specify className as instumentation filter args.

Expected Results

The gradle build should succeed as it run tests on the desired device factor, skipping tests only on the non-desired device.

Actual Results

even though the test run succeed, the gradle build failed

AndroidX Test and Android OS Versions

AGP - 8.5.1 Espresso - 3.6.1 androidXTestCore - 1.6.1 androidXTestJUnit - 1.2.1 androidXTestRules - 1.6.1 androidXTestRunner - 1.6.1 androidXTestOrchestrator - 1.5.0

Link to a public git repo demonstrating the problem:

kartik1o commented 2 months ago 
issue {
  namespace {
    namespace: "com.google.testing.platform.runtime.android.driver.AndroidInstrumentationDriver"
  }
  severity: SEVERE
  code: 1
  name: "INSTRUMENTATION_FAILED"
  message: "Instrumentation did not complete:\nInstrumentation code: 0\nInstrumentation result bundle: "
}

The above seems to be mentioned in the test-result.textproto file.