search

mozilla-mobile / firefox-tv

Firefox for Amazon's Fire TV
https://blog.mozilla.org/blog/2017/12/20/firefox-is-now-on-amazon-fire-tv-happy-holiday-watching/
Mozilla Public License 2.0
257 stars 111 forks source link

UI Tests cannot run in Debug mode #1569

Open nojunpark opened 5 years ago

nojunpark commented 5 years ago

Steps to reproduce

Run UI Test in Android Studio, to either device or simulator, in debug mode

Expected behavior

Test starts

Actual behavior

Test does not start, stuck after the command $ adb shell CLASSPATH=$(pm path android.support.test.services) app_process / android.support.test.services.shellexecutor.ShellMain am instrument -r -w -e targetInstrumentation org.mozilla.tv.firefox.debug.test/org.mozilla.tv.firefox.FirefoxTestRunner -e debug true -e class 'org.mozilla.tv.firefox.ui.BasicNavigationTest#basicNavigationTest' android.support.test.orchestrator/android.support.test.orchestrator.AndroidTestOrchestrator

Logcat shows this information:


2018-12-11 16:14:37.835 3599-3928/? I/ActivityManager: Force stopping android.support.test.orchestrator appid=10146 user=0: start instr
2018-12-11 16:14:37.835 3599-3928/? I/ActivityManager: Killing 24695:android.support.test.orchestrator/u0a146 (adj 0): stop android.support.test.orchestrator
2018-12-11 16:14:37.836 3599-3928/? D/ActivityManager: cleanUpApplicationRecord -- 24695
2018-12-11 16:14:37.836 3599-3928/? W/ActivityManager: Crash of app android.support.test.orchestrator running instrumentation ComponentInfo{android.support.test.orchestrator/android.support.test.orchestrator.AndroidTestOrchestrator}
2018-12-11 16:14:37.837 3599-3928/? I/ActivityManager: Force stopping android.support.test.orchestrator appid=10146 user=0: finished inst
2018-12-11 16:14:37.839 24678-24678/? D/AndroidRuntime: Shutting down VM
2018-12-11 16:14:37.858 3599-3928/? I/ActivityManager: Start proc 30590:android.support.test.orchestrator/u0a146 for added application android.support.test.orchestrator
2018-12-11 16:14:37.884 24667-24667/? I/art: System.exit called, status: 0
2018-12-11 16:14:37.884 24667-24667/? I/AndroidRuntime: VM exiting with result code 0.
2018-12-11 16:14:37.906 30590-30590/? W/System: ClassLoader referenced unknown path: /data/app/android.support.test.orchestrator-2/lib/arm
2018-12-11 16:14:37.907 30590-30590/? W/System: ClassLoader referenced unknown path: /system/priv-app/MetricsApi/lib/arm ```
nojunpark commented 5 years ago

I guess my question is why it's looking for /data/app/android.support.test.orchestrator-2 instead of /data/app/android.support.test.orchestrator . On other machines (where it still fails, I've seen it looking for /data/app/android.support.test.orchestrator-1 as well.

nojunpark commented 5 years ago

seems like Android Studio isn't the issue either, when adb shell am instrument -w -e debug true org.mozilla.tv.firefox.debug.test/org.mozilla.tv.firefox.FirefoxTestRunner is executed on cli, the test hangs as well. it runs test fine with other test options it seems.

nojunpark commented 5 years ago

This was mistakenly closed - the issue still exists, and with this bug UI tests cannot be debugged (i.e. setting a breakpoint, etc.)

nojunpark commented 5 years ago

There's a good reason to believe that this may be tied to the Test Orchestrator we're using - currently we use version 1.0.2., but using 1.1.0 still causes issue.

nojunpark commented 5 years ago

Further investigation shows that after commenting out execution 'ANDROID_TEST_ORCHESTRATOR' in gradle testOptions let you step through the test in debug mode.

Well, I suppose we can't get rid of test orchestrator at this point, @Baron-Severin (and other devs) how should we proceed?

severinrudie commented 5 years ago

TestOrchestrator is super important, but it only affects anything when we're running multiple tests at the same time (it causes each test to be run in its own process, to reduce shared state). I think we should be able to write some Gradle logic to detect the debugger flag and disable TestOrchestrator any time it's present. I usually only use the debugger when executing an individual test.

In the short term, I would recommend commenting out the execution 'ANDROID_TEST_ORCHESTRATOR' when debugging. That should be safe to do.

nojunpark commented 5 years ago

Agreed, closing this issue with above workaround

severinrudie commented 5 years ago

If you don't mind, I would actually like to keep it open until we make that Gradle change to fix it permanently. I'm hoping to find time to sneak it in in the next few days.

athomasmoz commented 5 years ago

@Baron-Severin did you add the permanent fix in?

severinrudie commented 5 years ago

@athomasmoz no, sorry, this one fell off my radar.

athomasmoz commented 5 years ago

No problem. Is the workaround sufficient or should the permanent fix be prioritized?

severinrudie commented 5 years ago

Workaround is fine for now. I'd call it maybe a P3? It's annoying to debug UI tests without the cleaner fix, but it's not a blocker.