Closed madept closed 1 year ago
Hey @madept.
I don't think your issue has anything to do with the adam vs ddmlib.
First, please migrate to the latest version 0.8.0, which has only adam. Second, please read the section of the docs on remote test parser and the requirements: https://docs.marathonlabs.io/android/configure#test-parser
From what you're describing above, you just added the
instrumentationArgs:
listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
without reading the docs first. Did you add the androidTestImplementation
scoped dependency as described? If you don't need the annotations (and I assume you don't even have them for cucumber) then just remove the listener.
I'm not sure specifying a non-existent listener is supported. The am instrument
execution is unlikely to even start unless listener exists in the classpath: the test parsing run likely just crashes without printing any tests, hence your no test cases found
error.
For context on why you see only non-cucumber-UI tests: spoon is rudimentary in the test scheduling, it starts unknown tests and just reacts to events coming from the test execution. Marathon on the other hand plans the test execution which means marathon has to get a list of tests expected to run first. Since the tests executed by cucumber are generated on-the-fly during execution remote test parser is the only option for you to get the list of expected tests to run. Bytecode analysis will not yield any results (this is what the default option using local test parser is using).
Thanks for the reply, I have already added following to gradle androidTestImplementation 'com.malinskiy.adam:android-junit4-test-annotation-producer:0.5.0'
regarding the document, it provides only these two things to be added which i already did.
I need the annotation and using kheera as runner. Also 0.8.0 does not seems to be in mavencentral yet. Can you please confirm ? latest version i see is 0.7.6 only
Error I am getting:
xxx-Next-4.57.0.0000-config-dev-androidTest.apk did not report any test annotations. If you need test annotations retrieval, remote test parser requires additional setup see https://marathonlabs.github.io/marathon/ven/android.html#test-parser
E 09:47:44.149 [main @coroutine#1]
I haven't worked with kheera, it was also last updated 5-6 years ago, and I'm not sure it supports the -e log true
which is essential for remote test parsing.
If your test run didn't report any test annotations with the additional lib then the run likely just crashes. To investigate if that's the case please run something like am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer com.example.test/androidx.test.runner.AndroidJUnitRunner
on the target Android device via adb shell replacing the test package com.example.test
and the test runner androidx.test.runner.AndroidJUnitRunner
with the ones used in your application. This should produce annotations similar to https://github.com/Malinskiy/adam/blob/master/android-junit4-test-annotation-producer/src/main/kotlin/com/malinskiy/adam/junit4/android/listener/TestAnnotationProducer.kt#L26. Maybe we can find a clue in the error of the command if you run it manually.
0.8.0 is out since yesterday: https://plugins.gradle.org/plugin/com.malinskiy.marathon. There is some migration from the old id though since it's now an official gradle plugin distributed using plugins.gradle.org
I tried following
adb shell am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer
..... INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: class=SomeClass INSTRUMENTATION_STATUS: numtests=39 INSTRUMENTATION_STATUS: test=Try to make a xyz Example No.1 ......
to my suprise. initially it gave process crashed and then i tried with -e perf true along with -e log true as per https://gist.github.com/tsohr/5711945 then it started providing metric , then i tried without -e perf true , then also it provided metric
After this i again tried running marathon and it failed, I tried to run it from the same command line where above metric is generated.
Any thing you can suggest
perf true is not a solution to the flakiness of your run. On the other hand, if it works with -e perf true
you can put those arguments into the test parsing command using instrumentationArgs
, e.g.:
testParserConfiguration:
type: "remote"
instrumentationArgs:
listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
perf: "true"
try if it helps. The output above looks correct, but I suspect that if this fails randomly then marathon is just not able to get a list of tests, hence your error
already tried did not work, how can i verify if com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer is getting triggered ? Or how to generate command line logs?
added debug : "true" to instrumentationArgs , getting following
D 17:07:17.088 [main @coroutine#1]
IDLE D 17:09:46.201 [main @coroutine#1] <c.m.m.android.adam.RemoteTestParser> Remote parsing failed. Retrying java.lang.RuntimeException: Unable to parse test list using 127.0.0.1:5037:RZ8NB114Q2Z at com.malinskiy.marathon.android.adam.RemoteTestParser.parseTests(RemoteTestParser.kt:116) at com.malinskiy.marathon.android.adam.RemoteTestParser.access$parseTests(RemoteTestParser.kt:37) at com.malinskiy.marathon.android.adam.RemoteTestParser$parseTests$1.invokeSuspend(RemoteTestParser.kt) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:33) at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279) at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85) at kotlinx.coroutines.BuildersKtBuildersKt.runBlocking(Builders.kt:59) at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) at kotlinx.coroutines.BuildersKtBuildersKt.runBlocking$default(Builders.kt:38) at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) at com.malinskiy.marathon.Marathon.run(Marathon.kt:52) at com.malinskiy.marathon.cli.ApplicationViewKt$main$1.invoke(ApplicationView.kt:68) at com.malinskiy.marathon.cli.ApplicationViewKt$main$1.invoke(ApplicationView.kt:30) at com.xenomachina.argparser.SystemExitExceptionKt.mainBody(SystemExitException.kt:74) at com.xenomachina.argparser.SystemExitExceptionKt.mainBody$default(SystemExitException.kt:72) at com.malinskiy.marathon.cli.ApplicationViewKt.main(ApplicationView.kt:28)
You can verify the listener is triggered by looking on the am instrument output. It should contain annotations, e.g.
INSTRUMENTATION_STATUS: com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer.v2=[androidx.test.filters.SmallTest(), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow), org.junit.Test(expected=class org.junit.Test$None:timeout=0), io.qameta.allure.kotlin.Owner(value=user2), io.qameta.allure.kotlin.Feature(value=Text on main screen), io.qameta.allure.kotlin.Epic(value=General), org.junit.runner.RunWith(value=class io.qameta.allure.android.runners.AllureAndroidJUnit4), kotlin.Metadata(bytecodeVersion=[I@bdf6b25:data1=[Ljava.lang.String;@46414fa:data2=[Ljava.lang.String;@5d4aab:extraInt=0:extraString=:kind=1:metadataVersion=[I@fbb1508:packageName=), io.qameta.allure.kotlin.Severity(value=critical), io.qameta.allure.kotlin.Story(value=Slow)]
This should be in the output of adb shell am instrument -w -r -e log true -e listener com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer .test/.runner.KheeraJUnitRunner
, the command you've used above
Thanks for the reply,
testParserConfiguration = new TestParserConfiguration.RemoteTestParserConfiguration(["listener":"com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"])
when i am trying 0.8.1 plugin, getting TestParserConfiguration not found error. What to add to dependency for this ? Or how to add remote parser to plugin configuration in gradle ?
Also while running from command line , i am getting following error now
W 13:56:37.917 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#443] < TestRunResultsListener > uncompleted = Perform
To investigate further I tried to run directly following command , I get error No instrumentation found ./adb shell am instrument -w -r --no-window-animation -e class Perform\ Join\ Tests#Customer\ start\ JOIN\ flow\ successfully < appid >/< runner > onError: commandError=true message=No instrumentation found for: Join In the above i got output am instrument -w -r --no-window-animation -e class Perform\ Join\ Tests#Customer\ start\ JOIN\ flow\ successfully < appid >/< runner > From marathon command line where I am getting incomplete, Also I tried providing feature file in instrumentation args. Please help, seems i am close to solve it
@Malinskiy If I pass additional argument to testparserconfiguraiton instrumentationArgs or normal Args, seems it is not passed to am instrument, how can i verify that ?
For the test args - they're printed to output with debug=true. For parsing your only option to verify this is debug marathon's code.
I really doubt you have any problems with passing args: its a stable feature for a long time and is covered with tests. Might be misconfiguration or some other problem.
Are you able to parse tests now or is this still related to the parsing?
@Malinskiy , I was able to resolve everything, i have a final query if you can help with, Query is
In gradle( groovy) file i have to use following
var dryRunArgs =[listener:"com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"] testParserConfiguration = new TestParserConfiguration.RemoteTestParserConfiguration(dryRunArgs) For this I had to use import in groovy is it possible to provide testParserConfigration like this testParserConfiguration{ parserType(or something): "remote" instrumentationArgs: map<String, String> }
please suggest? Help is appreciated, once you provide input, I will close this bug. Thanks for all the support
Currently this is not supported and highly likely will not be supported: selecting the parser via enum is much safer than via a string since it fails before the run during compilation
Kts provides autocompletion for these enums btw if you want more help from your IDE on filling those fields.
@Malinskiy thanks for the reply, It is done in Marathonfile that way only, and finally gradle plugin is generating marathon file only and run it. is that not true ? So wont it be only generating marathonfile using these input ? Sorry but just curious !!
You are correct that gradle plugin generates the marathonfile.
If we were to support untyped parameters during configuration via gradle plugin then the only place where they will be verified will be when the cli version of marathon is triggered. This way if something is wrong you can commit a configuration that will fail in runtime, i.e. only when tests are executed.
Gradle plugin also supports just generating the marathonfile without running tests, running such generation will produce an invalid marathonfile if the type will be just a string.
In short, the single source of truth for the configuration are the classes, not their representation via yaml/gradle dsl. The more you use real source of truth - the less chance you have of an invalid configuration.
I seems to be working fine , closing bug now
Hi,
I am getting following exception while using adam as vendor
Here is my marathonfile Generated from gradle plugin. I am using gradle plugin and my test cases are cucumber bases, earlier I was using spoon and it was working fine.
Also when i use ddmlib, it runs non UI test only. Please help me to migrate to marathon
Marathonfile