search

gojuno / composer

Reactive Android Instrumentation Test Runner. Archived. Marathon is recommended as an alternative (https://github.com/Malinskiy/marathon).
Apache License 2.0
544 stars 45 forks source link

Add Test Orchestrator support. #90

Closed artem-zinnatullin closed 6 years ago

artem-zinnatullin commented 7 years ago

You can trick Composer to work with Orchestrator right now, but more official solution would be better.

Also, as we've measured in #66 instrumenting each test separately can add 3-5 seconds overhead for each test which can be unacceptable for those who run them frequently (each PR/commit), but we need to actually measure this with Orchestrator.

See:

artem-zinnatullin commented 7 years ago

@ming13 any additions here since you've tried Orchestrator? I am also curious about test run duration in compare to regular am instrumentation.

arturdryomov commented 7 years ago

Okay, I’m doing everything manually from the command line.

$ adb install -r app.apk
$ adb install -r test.apk

$ adb install -r orchestrator.apk
$ adb install -r test-services.apk

Just instrument

adb shell "am instrument -r -w \
  -e package com.gojuno.actually.some.package \
  com.gojuno.rider.test/android.support.test.runner.JunoAndroidRunner"

Orchestrator

adb shell 'CLASSPATH=$(pm path android.support.test.services) app_process / \
  android.support.test.services.shellexecutor.ShellMain am instrument -r -w \
  -e package com.gojuno.actually.some.package \
  -e targetInstrumentation com.gojuno.rider.test/android.support.test.runner.JunoAndroidRunner \
  android.support.test.orchestrator/.AndroidTestOrchestrator'

BTW.

-e numShards and -e shardIndex runner arguments aren't currently supported

Links:

arturdryomov commented 7 years ago

I’ll file the issue about missing instrument output (maybe) tomorrow.

artem-zinnatullin commented 7 years ago

Very interesting to see that it took THAT less time. I suspect that not all tests were run

artem-zinnatullin commented 7 years ago

Great, please crosslink this issue with instrumentation output issue in Google bugtracker!

arturdryomov commented 7 years ago

Very interesting to see that it took THAT less time. I suspect that not all tests were run

I actually counted them visually and they all were here.

arturdryomov commented 7 years ago

Here we go — https://issuetracker.google.com/issues/64486480

artem-zinnatullin commented 7 years ago

You probably better copy "steps to reproduce" there as well

handstandsam commented 7 years ago

@artem-zinnatullin - You mentioned there is a way to currently trick composer into using the test orchestrator? What is that trick?

I have been able to run my tests with the orchestrator via adb shell like @ming13 did, but haven't figured out how to do it via Composer. Thanks!

artem-zinnatullin commented 7 years ago

Unfortunately since adb shell am command is really different with Orchestrator there is no way to "trick" Composer to use Orchestrator, also it's unclear from https://issuetracker.google.com/issues/64486480 and release notes if they fixed instrumentation output of Orchestrator or not.

Once they fix it, we can start implementing support for Orchestrator :)

yunikkk commented 7 years ago

I've just tried the latest version of test libs and discovered that it writes xml reports to the sdcard /sdcard/odo/*, eg.

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="6" failures="0" errors="0" skipped="1" time="25.982" timestamp="2017-10-25T23:22:20" hostname="localhost">
  <properties />
  <testcase name="testAboutAndCheckUpdates" classname="com.gojuno.driver.robotPattern.tests.menu.about.AboutUITest" start-time="1508973716.203" end-time="1508973718.141" time="1.938" />
  <testcase name="testCommonHelp" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForAuthorizedUserUITest" start-time="1508973720.870" end-time="1508973723.102" time="2.232" />
  <testcase name="testMoreOptions" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForAuthorizedUserUITest" start-time="1508973725.621" end-time="1508973728.267" time="2.646" />
  <testcase name="testHelpContactDataAreDisplayed" classname="com.gojuno.driver.robotPattern.tests.menu.account.help.HelpForUnauthorizedUserUITest" start-time="1508973730.770" end-time="1508973731.869" time="1.099" />
  <testcase name="testErrorOnCheckForUpdatesIsDisplayed" classname="com.gojuno.driver.robotPattern.tests.unauthorized.LoginWithOldAppVersionTest" start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9">
    <skipped />
  </testcase>
  <testcase name="testADSHardBlockScreenIsDisplayed" classname="com.gojuno.driver.robotPattern.tests.unauthorized.LoginWithOldAppVersionTest" start-time="1508973736.861" end-time="1508973739.655" time="2.794" />
</testsuite>
artem-zinnatullin commented 7 years ago

Well we can hook into that, but that won't be "reactive", instrumentation output is printed asap, but I bet JUnit report only appears after test run

start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9"

lol wut

yunikkk commented 7 years ago

Yup, reports appears only after run, checked that. Also, sharding seems to be working, too.

yunikkk commented 7 years ago

start-time="1508973734.374" end-time="0.000" time="-1.508973734374E9" its for ignored test =)

originx commented 6 years ago

any news on this issue, is it possible to implement it with current version of orchestrator.

Btw have you ever considered JUnit listeners, or piggybacking on existing ones: https://android.googlesource.com/platform/frameworks/testing/+/3352c37/androidtestlib/src/com/android/test/runner/listener/

?

yunikkk commented 6 years ago

Well, I haven't tried since October, will need to recheck status of issues we faced.

Regarding listeners - to use them we'll need to run tests with our own customized TestRunner, I guess. That will broke cases for the users who implements their own runners in the test apk. Not sure if there is a workaround... @artem-zinnatullin wdyt about listeners & runner?

originx commented 6 years ago

you can add the listeners to the existing test runner: To specify one or more RunListeners to observe the test run: -e listener com.foo.Listener,com.foo.Listener2 https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner.html

there is also a possibility to add them in manifest or via gradle configuration ie. instrumentation arguments.

or in the end you can go like spoon did and provide the customized testrunner via the gradle plugin: https://github.com/square/spoon/tree/master/spoon-runner/src/main/java/com/squareup/spoon

but I think it is a safer way to go compared to the bash output from the instrumentation process?

trevjonez commented 6 years ago

even if they use the listener interface they still need to communicate back to the host machine for report generation. parsing instrumentation process output is tedious and perhaps fragile from the maintenance perspective but the next obvious communication medium for streaming the data back to the host would be some sort of network connection from the device which can probably be statistically proven as a large source of flake if any data was tracked/analyzed.

perhaps if not a network connection the listener would write to a file on device that could then be tailed over adb on the host?

Other than being able to control the communication format I don't see a huge value to doing it that way.

artem-zinnatullin commented 6 years ago

Yay, I don't see benefit in using listeners right now. Orchestrator needs to be reinvestigated, last time we checked it didn't produce correct instrumentation output.

Alternative could be to have own "Orchestrator", but I'd prefer not to do this.

Another alternative could be to start test run that runs single test, stops the app and clears its date. Do it for every test and you have Orchestrator. It can be done on top of existing logic, we'll just need to parse apk to get test classes and methods. The only problem with this solution is that it'll work well only with JUnit based tests.

On Thu, Mar 8, 2018, 8:33 AM Trevor Jones notifications@github.com wrote:

even if they use the listener interface they still need to communicate back to the host machine for report generation. parsing instrumentation process output is tedious and perhaps fragile from the maintenance perspective but the next obvious communication medium for streaming the data back to the host would be some sort of network connection from the device which can probably be statistically proven as a large source of flake if any data was tracked/analyzed.

perhaps if not a network connection the listener would write to a file on device that could then be tailed over adb on the host?

Other than being able to control the communication format I don't see a huge value to doing it that way.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gojuno/composer/issues/90#issuecomment-371542614, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3KlR1i-OimAZwd-g1lEwauWwi2XDks5tcV1jgaJpZM4Ok3Va .

evgengal commented 6 years ago

Any news on this issue?

artem-zinnatullin commented 6 years ago

Last related PR: #138

However Google is working on Nitrogen that overlaps with Composer functionality which might affect our plans

evgengal commented 6 years ago

Do you mean Nitrogen OS?

dsvoronin commented 6 years ago

Nope, "Project nitrogen" announced on IO https://www.youtube.com/watch?v=wYMIadv9iF8

CristianGM commented 6 years ago

@artem-zinnatullin If the year ends without Google releasing Nitrogen ( it wouldn't be the first time, thats why usually they don't say dates ) all devs using Orchestrator will have wasted a year without Composer ( so, having a good sharding and report ).

I don't get the answer saying Nitrogen might affect Composer plans, is like abandoning the project just because eventually there will be something better.