search

gojuno / composer

Reactive Android Instrumentation Test Runner. Archived. Marathon is recommended as an alternative (https://github.com/Malinskiy/marathon).
Apache License 2.0
546 stars 45 forks source link

Make sure Composer marks test run as failed if test runner throws error before actual test run. #89

Open artem-zinnatullin opened 7 years ago

artem-zinnatullin commented 7 years ago

AndroidJUnitRunner from test support library v1.0 fails test run if app or test apk references non-existing classes. See https://issuetracker.google.com/u/1/issues/64094195

Runner v0.5 printed that as instrumentation problem but continued to run tests (that was not perfect, but better behavior), Runner v1.0 fails test run.

Firebase Test Cloud happily marks test run as successful in both cases even though in fact in second case none of the tests did actually run.

That's why I'm creating this issue against Composer to make sure (add test) that following instrumentation output will cause error in Composer and test run won't be marked as successful.

INSTRUMENTATION_RESULT: stream=

Time: 0
There was 1 failure:
1) Fatal exception when running tests
java.lang.NoClassDefFoundError: Failed resolution of: Ljavax/annotation/processing/AbstractProcessor;
    at java.lang.Class.classForName(Native Method)
    at java.lang.Class.forName(Class.java:400)
    at android.support.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:76)
    at android.support.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
    at android.support.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:808)
    at android.support.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:481)
    at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:367)
    at com.app.test.InstrumentationRunner.onStart(InstrumentationRunner.java:26)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1962)
Caused by: java.lang.ClassNotFoundException: Didn't find class "javax.annotation.processing.AbstractProcessor" on path: DexPathList[[zip file "/system/framework/android.test.runner.jar", zip file "/data/app/com.app.test-1/base.apk", zip file "/data/app/com.app-1/base.apk"],nativeLibraryDirectories=[/data/app/com.app.test-1/lib/arm64, /data/app/com.app-1/lib/arm, /system/fake-libs, /data/app/com.app.test-1/base.apk!/lib/armeabi-v7a, /data/app/com.app-1/base.apk!/lib/armeabi-v7a, /system/lib, /vendor/lib]]
    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:380)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    ... 9 more

FAILURES!!!
Tests run: 0,  Failures: 1

INSTRUMENTATION_CODE: -1

Composer already fails test run if 0 tests were run, but I just want make code even more robust against such stupid errors and be better than Firebase at parsing Google's crazy instrumentation output format.