No Test cases found while using adam as vendor

[madept]

Hi,

I am getting following exception while using adam as vendor

If you need test annotations retrieval, remote test parser requires additional setup see https://marathonlabs.github.io/marathon/ven/android.html#test-parser
E 05:42:14.023 [main @coroutine#1] <com.malinskiy.marathon.Marathon> com.malinskiy.marathon.exceptions.NoTestCasesFoundException: No tests cases were found

Here is my marathonfile Generated from gradle plugin. I am using gradle plugin and my test cases are cucumber bases, earlier I was using spoon and it was working fine.

Also when i use ddmlib, it runs non UI test only. Please help me to migrate to marathon

Marathonfile

name: "Marathon Tests"
outputDir: "/Users/kumark56/.gradle/daemon/7.4.2/build/reports/marathon/configDevAndroidTest"
outputConfiguration:
  maxPath: 1024
analyticsConfiguration:
  type: "disabled"
poolingStrategy:
  type: "omni"
shardingStrategy:
  type: "count"
  count: 5
sortingStrategy:
  type: "no-sorting"
batchingStrategy:
  type: "isolate"
flakinessStrategy:
  type: "ignore"
retryStrategy:
  type: "fixed-quota"
  totalAllowedRetryQuota: 100
  retryPerTestQuota: 3
filteringConfiguration:
  allowlist: []
  blocklist: []
ignoreFailures: false
isCodeCoverageEnabled: false
fallbackToScreenshots: false
strictMode: false
uncompletedTestRetryQuota: 100
testClassRegexes:
- ".*"
includeSerialRegexes: []
excludeSerialRegexes: []
testBatchTimeoutMillis: 1800000
testOutputTimeoutMillis: 130000
debug: true
screenRecordingPolicy: "ON_FAILURE"
vendorConfiguration:
  type: "Android"
  vendor: "DDMLIB"
  androidSdk: "/Users/kumark56/Library/Android/sdk"
  outputs:
  - application: "<path>/xxx-Next-4.57.0.0000-config-dev.apk"
    testApplication: "<path>/xxx-Next-4.57.0.0000-config-dev-androidTest.apk"
  autoGrantPermission: true
  instrumentationArgs: {}
  applicationPmClear: false
  testApplicationPmClear: false
  adbInitTimeoutMillis: 30000
  installOptions: ""
  serialStrategy: "AUTOMATIC"
  screenRecordConfiguration:
    videoConfiguration:
      enabled: true
      width: 720
      height: 1280
      bitrateMbps: 1
      timeLimit: 180
      timeLimitUnits: "SECONDS"
    screenshotConfiguration:
      enabled: true
      width: 720
      height: 1280
      delayMs: 500
  waitForDevicesTimeoutMillis: 30000
  allureConfiguration:
    enabled: false
    relativeResultsDirectory: "/files/allure-results"
    pathRoot: "APP_DATA"
  timeoutConfiguration:
    shell: 20.000000000
    listFiles: 20.000000000
    pushFile: 60.000000000
    pushFolder: 60.000000000
    pullFile: 30.000000000
    uninstall: 20.000000000
    install: 20.000000000
    screenrecorder: 200.000000000
    screencapturer: 0.300000000
    socketIdleTimeout: 30.000000000
    portForward: 20.000000000
  fileSyncConfiguration:
    pull: []
    push: []
  threadingConfiguration:
    bootWaitingThreads: 4
    adbIoThreads: 4
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
  testAccessConfiguration:
    adb: false
    grpc: false
    console: false
    consoleToken: ""
  adbServers:
  - host: "127.0.0.1"
    port: 5037
  disableWindowAnimation: true
analyticsTracking: false
deviceInitializationTimeoutMillis: 180000

[Malinskiy]

Hey @madept.

I don't think your issue has anything to do with the adam vs ddmlib.

First, please migrate to the latest version 0.8.0, which has only adam. Second, please read the section of the docs on remote test parser and the requirements: https://docs.marathonlabs.io/android/configure#test-parser

From what you're describing above, you just added the

    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"

without reading the docs first. Did you add the androidTestImplementation scoped dependency as described? If you don't need the annotations (and I assume you don't even have them for cucumber) then just remove the listener.

I'm not sure specifying a non-existent listener is supported. The am instrument execution is unlikely to even start unless listener exists in the classpath: the test parsing run likely just crashes without printing any tests, hence your no test cases found error.

For context on why you see only non-cucumber-UI tests: spoon is rudimentary in the test scheduling, it starts unknown tests and just reacts to events coming from the test execution. Marathon on the other hand plans the test execution which means marathon has to get a list of tests expected to run first. Since the tests executed by cucumber are generated on-the-fly during execution remote test parser is the only option for you to get the list of expected tests to run. Bytecode analysis will not yield any results (this is what the default option using local test parser is using).

[madept]

Thanks for the reply, I have already added following to gradle androidTestImplementation 'com.malinskiy.adam:android-junit4-test-annotation-producer:0.5.0'

regarding the document, it provides only these two things to be added which i already did.

I need the annotation and using kheera as runner. Also 0.8.0 does not seems to be in mavencentral yet. Can you please confirm ? latest version i see is 0.7.6 only

Error I am getting:

xxx-Next-4.57.0.0000-config-dev-androidTest.apk did not report any test annotations. If you need test annotations retrieval, remote test parser requires additional setup see https://marathonlabs.github.io/marathon/ven/android.html#test-parser E 09:47:44.149 [main @coroutine#1] com.malinskiy.marathon.exceptions.NoTestCasesFoundException: No tests cases were found

[Malinskiy]

I haven't worked with kheera, it was also last updated 5-6 years ago, and I'm not sure it supports the -e log true which is essential for remote test parsing.

If your test run didn't report any test annotations with the additional lib then the run likely just crashes. To investigate if that's the case please run something like am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer com.example.test/androidx.test.runner.AndroidJUnitRunner on the target Android device via adb shell replacing the test package com.example.test and the test runner androidx.test.runner.AndroidJUnitRunner with the ones used in your application. This should produce annotations similar to https://github.com/Malinskiy/adam/blob/master/android-junit4-test-annotation-producer/src/main/kotlin/com/malinskiy/adam/junit4/android/listener/TestAnnotationProducer.kt#L26. Maybe we can find a clue in the error of the command if you run it manually.

0.8.0 is out since yesterday: https://plugins.gradle.org/plugin/com.malinskiy.marathon. There is some migration from the old id though since it's now an official gradle plugin distributed using plugins.gradle.org

[madept]

I tried following adb shell am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer .test/.runner.KheeraJUnitRunner and it produces following on command line

..... INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: class=SomeClass INSTRUMENTATION_STATUS: numtests=39 INSTRUMENTATION_STATUS: test=Try to make a xyz Example No.1 ......

to my suprise. initially it gave process crashed and then i tried with -e perf true along with -e log true as per https://gist.github.com/tsohr/5711945 then it started providing metric , then i tried without -e perf true , then also it provided metric

After this i again tried running marathon and it failed, I tried to run it from the same command line where above metric is generated.

Any thing you can suggest

[Malinskiy]

perf true is not a solution to the flakiness of your run. On the other hand, if it works with -e perf true you can put those arguments into the test parsing command using instrumentationArgs, e.g.:

  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
      perf: "true"

try if it helps. The output above looks correct, but I suspect that if this fails randomly then marathon is just not able to get a list of tests, hence your error

[madept]

already tried did not work, how can i verify if com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer is getting triggered ? Or how to generate command line logs?

[madept]

added debug : "true" to instrumentationArgs , getting following

D 17:07:17.088 [main @coroutine#1] Android Debug Bridge /127.0.0.1:5037: version 41

IDLE D 17:09:46.201 [main @coroutine#1] <c.m.m.android.adam.RemoteTestParser> Remote parsing failed. Retrying java.lang.RuntimeException: Unable to parse test list using 127.0.0.1:5037:RZ8NB114Q2Z at com.malinskiy.marathon.android.adam.RemoteTestParser.parseTests(RemoteTestParser.kt:116) at com.malinskiy.marathon.android.adam.RemoteTestParser.access$parseTests(RemoteTestParser.kt:37) at com.malinskiy.marathon.android.adam.RemoteTestParser$parseTests$1.invokeSuspend(RemoteTestParser.kt) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:33) at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279) at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85) at kotlinx.coroutines.BuildersKtBuildersKt.runBlocking(Builders.kt:59) at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) at kotlinx.coroutines.BuildersKtBuildersKt.runBlocking$default(Builders.kt:38) at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) at com.malinskiy.marathon.Marathon.run(Marathon.kt:52) at com.malinskiy.marathon.cli.ApplicationViewKt$main$1.invoke(ApplicationView.kt:68) at com.malinskiy.marathon.cli.ApplicationViewKt$main$1.invoke(ApplicationView.kt:30) at com.xenomachina.argparser.SystemExitExceptionKt.mainBody(SystemExitException.kt:74) at com.xenomachina.argparser.SystemExitExceptionKt.mainBody$default(SystemExitException.kt:72) at com.malinskiy.marathon.cli.ApplicationViewKt.main(ApplicationView.kt:28)

[Malinskiy]

You can verify the listener is triggered by looking on the am instrument output. It should contain annotations, e.g.

INSTRUMENTATION_STATUS: com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer.v2=[androidx.test.filters.SmallTest(), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow), org.junit.Test(expected=class org.junit.Test$None:timeout=0), io.qameta.allure.kotlin.Owner(value=user2), io.qameta.allure.kotlin.Feature(value=Text on main screen), io.qameta.allure.kotlin.Epic(value=General), org.junit.runner.RunWith(value=class io.qameta.allure.android.runners.AllureAndroidJUnit4), kotlin.Metadata(bytecodeVersion=[I@bdf6b25:data1=[Ljava.lang.String;@46414fa:data2=[Ljava.lang.String;@5d4aab:extraInt=0:extraString=:kind=1:metadataVersion=[I@fbb1508:packageName=), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow)]

This should be in the output of adb shell am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer .test/.runner.KheeraJUnitRunner, the command you've used above

[madept]

Thanks for the reply,

testParserConfiguration = new TestParserConfiguration.RemoteTestParserConfiguration(["listener":"com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"])

when i am trying 0.8.1 plugin, getting TestParserConfiguration not found error. What to add to dependency for this ? Or how to add remote parser to plugin configuration in gradle ?

Also while running from command line , i am getting following error now

W 13:56:37.917 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#443] < TestRunResultsListener > uncompleted = Perform Tests#, 127.0.0.1:5037:emulator-5554

To investigate further I tried to run directly following command , I get error No instrumentation found ./adb shell am instrument -w -r --no-window-animation -e class Perform\ Join\ Tests#Customer\ start\ JOIN\ flow\ successfully < appid >/< runner > onError: commandError=true message=No instrumentation found for: Join In the above i got output am instrument -w -r --no-window-animation -e class Perform\ Join\ Tests#Customer\ start\ JOIN\ flow\ successfully < appid >/< runner > From marathon command line where I am getting incomplete, Also I tried providing feature file in instrumentation args. Please help, seems i am close to solve it

[madept]

@Malinskiy If I pass additional argument to testparserconfiguraiton instrumentationArgs or normal Args, seems it is not passed to am instrument, how can i verify that ?

[Malinskiy]

For the test args - they're printed to output with debug=true. For parsing your only option to verify this is debug marathon's code.

I really doubt you have any problems with passing args: its a stable feature for a long time and is covered with tests. Might be misconfiguration or some other problem.

Are you able to parse tests now or is this still related to the parsing?

[madept]

@Malinskiy , I was able to resolve everything, i have a final query if you can help with, Query is

In gradle( groovy) file i have to use following

var dryRunArgs =[listener:"com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"] testParserConfiguration = new TestParserConfiguration.RemoteTestParserConfiguration(dryRunArgs) For this I had to use import in groovy is it possible to provide testParserConfigration like this testParserConfiguration{ parserType(or something): "remote" instrumentationArgs: map<String, String> }

please suggest? Help is appreciated, once you provide input, I will close this bug. Thanks for all the support

[Malinskiy]

Currently this is not supported and highly likely will not be supported: selecting the parser via enum is much safer than via a string since it fails before the run during compilation

Kts provides autocompletion for these enums btw if you want more help from your IDE on filling those fields.

[madept]

@Malinskiy thanks for the reply, It is done in Marathonfile that way only, and finally gradle plugin is generating marathon file only and run it. is that not true ? So wont it be only generating marathonfile using these input ? Sorry but just curious !!

[Malinskiy]

You are correct that gradle plugin generates the marathonfile.

If we were to support untyped parameters during configuration via gradle plugin then the only place where they will be verified will be when the cli version of marathon is triggered. This way if something is wrong you can commit a configuration that will fail in runtime, i.e. only when tests are executed.

Gradle plugin also supports just generating the marathonfile without running tests, running such generation will produce an invalid marathonfile if the type will be just a string.

In short, the single source of truth for the configuration are the classes, not their representation via yaml/gradle dsl. The more you use real source of truth - the less chance you have of an invalid configuration.

[madept]

I seems to be working fine , closing bug now