Instrumented test execution with ANDROIDX_TEST_ORCHESTRATOR does not work with Android 11

mateuszkwiecinski commented 4 years ago

Description

Similar to #355, but the execution hangs

Steps to Reproduce

Create new project using AS wizard
Enable Android orchestrator following the https://developer.android.com/training/testing/junit-runner
Run tests on Android Emulator with API 30

Expected Results

The test passes

Actual Results

The test hangs

logcat shows:

2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.myapplication, PID: 12791
java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.myapplication.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709)
    at android.app.ActivityThread.access$1300(ActivityThread.java:237)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loop(Looper.java:223)
    at android.app.ActivityThread.main(ActivityThread.java:7656)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
 Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService
    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.connect(OrchestratedInstrumentationListener.java:93)
    at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:309)
    at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6704)
    at android.app.ActivityThread.access$1300(ActivityThread.java:237) 
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) 
    at android.os.Handler.dispatchMessage(Handler.java:106) 
    at android.os.Looper.loop(Looper.java:223) 
    at android.app.ActivityThread.main(ActivityThread.java:7656) 
    at java.lang.reflect.Method.invoke(Native Method) 
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) 
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) 
2020-09-15 21:28:39.486 12791-12791/com.example.myapplication E/AndroidJUnitRunner: An unhandled exception was thrown by the app.
2020-09-15 21:28:39.493 12791-12791/com.example.myapplication E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash

(might be unrelated, but doesn' happen on api 29) my private app shows in addition: mkdir: ‘/storage/emulated/0/Android/data/my.app.package’: Permission denied which points at DeviceProviderInstrumentTestTask and getAdditionalTestOutputEnabled which tries to save some log file. disabling the android.enableAdditionalTestOutput property removes the mkdir error but the test still hangs

AndroidX Test and Android OS Versions

    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'androidx.test:runner:1.3.0'
    androidTestUtil 'androidx.test:orchestrator:1.3.0'

Android OS Versions: the issue exists only on API 30. The test passes on lower apis.

Link to a public git repo demonstrating the problem:

https://github.com/mateuszkwiecinski/orchestrator_doesnt_work task: ./gradlew connectedDebugAndroidTest hangs and never ends

Github action run confirming the issue: https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/1/checks?check_run_id=1119649523

prfarlow1 commented 4 years ago

I am also experiencing this bug. I also made a test application that experiences the same indefinite hanging on Android 11: https://github.com/prfarlow1/UiTestApplication

LmKupke commented 4 years ago

Same issue is occurring when compiling and targeting SDK 30 and running against an emulator with 30.

2020-09-21 14:26:44.942 31419-31419/com.scompany.consumer E/AndroidRuntime: FATAL EXCEPTION: main Process: com.scompany.consumer, PID: 31419 java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6709) at android.app.ActivityThread.access$1300(ActivityThread.java:237) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:223) at android.app.ActivityThread.main(ActivityThread.java:7656) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947) Caused by: java.lang.RuntimeException: Cannot connect to androidx.test.orchestrator.OrchestratorService

no issue when compiling and targeting 30 and running on an emulator with 29.
no issue when compiling and targeting 29 and running on an emulator with 30.

prfarlow1 commented 4 years ago

Can we get an update from the Android test team when this will be addressed? This is a really bad bug. We rely on the orchestrator for our tests to pass and right now we just can't run our tests on Android 11 test devices.

nimeacuerdo commented 4 years ago

Same here. Everything was working fine with emulator api 29 and Orchestrator until we migrated to api 30.

brettchabot commented 4 years ago

Hey all, sorry for delayed response. We are looking into this, but the fix is proving more difficult to get released than anticipated. We hope to have something out soon to address this.

tillchen commented 4 years ago

Same here. Switched to Android 10 and it's working fine for me.

brettchabot commented 4 years ago

Please give 1.3.1-alpha01 a try. This should hopefully be fixed. See https://github.com/android/android-test/releases/tag/androidx-test-1.3.1-alpha01

mateuszkwiecinski commented 4 years ago

Can't confirm it works :/ It doesn't hang indeed, but still fails with a message:

Task :app:connectedDebugAndroidTest Starting 0 tests on test(AVD) - 11 Tests on test(AVD) - 11 failed: Instrumentation run failed due to 'Process crashed.'

com.android.build.gradle.internal.testing.ConnectedDevice > No tests found.[test(AVD) - 11] FAILED No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack @Test annotations).

Failed run: https://github.com/mateuszkwiecinski/orchestrator_doesnt_work/pull/3/checks?check_run_id=1274143298

It works on devices with api <30

prfarlow1 commented 4 years ago

I also could not get my tests to work with the new 1.3.1-alpha01

brettchabot commented 4 years ago

It looks like there is an incompatibility with the testInstrumentationRunnerArguments clearPackageData: 'true' setting. I was able to get @mateuszkwiecinski sample working by commenting out that line.

brettchabot commented 4 years ago

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

prfarlow1 commented 4 years ago

A workaround for the clearPackageData issue is to declare a

application android:forceQueryable="true"

in the androidTest manifest

What does android:forceQueryable="true" actually do? I can't find any documentation for it

brettchabot commented 4 years ago

Check out the docs on the new package visibility restrictions on API 30 here https://developer.android.com/training/basics/intents/package-visibility

IIUC using forceQueryable basically lets other apps (like the orchestrator app) communicate with the test package.

brettchabot commented 4 years ago

Please give 1.3.1-alpha02 a try

mateuszkwiecinski commented 4 years ago

Thank you @brettchabot 👍 I can confirm it works in my sample project 🚀

brettchabot commented 4 years ago

whew thanks for confirming @mateuszkwiecinski

tadfisher commented 4 years ago

android:forceQueryable isn't recognized by AAPT2 4.1.0 and fails the build:

Execution failed for task ':Mobile-Android:processDevDebugAndroidTestResources'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > Android resource linking failed
     /home/tad/simple/mobile/Mobile-Android/Mobile-Android/build/intermediates/packaged_manifests/devDebugAndroidTest/AndroidManifest.xml:22: AAPT: error: attribute android:forceQueryable not found.

brettchabot commented 4 years ago

@tadfisher can you be more specific about your project setup? I don't know where a 'AAPT2 4.1.0' comes from.

The aapt2 version in build tools 30.0.2 appears to be ~/Android/Sdk/build-tools/30.0.2$ ./aapt2 version Android Asset Packaging Tool (aapt) 2.19-6739378

tadfisher commented 4 years ago

@brettchabot My bad, this is on a nonstandard setup using aapt2 from maven.google.com. With the aapt2 packaged with build-tools 30.0.2, this works.

brettchabot commented 4 years ago

@tadfisher cool thanks for confirming.

mateuszkwiecinski commented 4 years ago

@brettchabot Just in case if that's related: I faced another issue with 1.3.1-alpha02 - our tests don't fail anymore. Apparently new alpha version changes when org.junit.runner.notification.RunListener callbacks are called, which breaks LeakCanary, which throws an exception and for an unknown reason makes the test to pass. I reported an issue there (https://github.com/square/leakcanary/issues/1986) but as well that might be an unexpected behavior change in the android-test library

C2H6O commented 3 years ago

@brettchabot after upgrading to alpha02, seeing the following crash:

Process: com.plangrid.android.devgrid, PID: 29601
    java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.plangrid.android.devgrid.test/com.plangrid.android.CustomAndroidJUnitRunner}: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6712)
        at android.app.ActivityThread.access$1300(ActivityThread.java:237)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1913)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7660)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
     Caused by: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService
        at androidx.test.internal.events.client.TestEventServiceConnectionBase.connect(TestEventServiceConnectionBase.java:94)
        at androidx.test.internal.events.client.TestEventClient.connect(TestEventClient.java:125)
        at androidx.test.runner.AndroidJUnitRunner.isOrchestratorServiceProvided(AndroidJUnitRunner.java:343)
        at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:315)

najmsheikh commented 3 years ago

Hi @brettchabot, we cannot migrate our project to AndroidX just yet so we would need to use the Support test runner. Our project is setup like so:

android {
    defaultConfig {
        minSdkVersion 19
        targetSdkVersion 28
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        testInstrumentationRunnerArguments clearPackageData: 'true'
    }
    testOptions {
        execution 'ANDROID_TEST_ORCHESTRATOR'
    }
}
dependencies {
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test:rules:1.0.2'
    androidTestUtil 'com.android.support.test:orchestrator:1.0.2'
}

Running the instrumentation test suite -- either via Android Studio or Gradle -- results in a "No Tests Found" error when the connected device is running Android 10 or above.

Is there a solution available that allows us to clear state between tests while not requiring us to upgrade to AndroidX?

brettchabot commented 3 years ago

No sorry I'm not aware of a workaround when using the com.android.support.test packages.

TWiStErRob commented 3 years ago

@najmsheikh during migration, we had an intermediate state, when we used AndroidX Espresso and runner, but still used support libs in production code. So that is possible, maybe it helps you. No need for Jetifier, just enable useAndroidX to make it build in AGP.

najmsheikh commented 3 years ago

@TWiStErRob That's quite interesting. I was under the impression that you could not have both Support Library and AndroidX dependencies in the same project (even if they're different modules). Did you have to do anything special to get the both to coexist? In any case, thanks for the suggestion. I'll try and report back here for anyone else following along.

brettchabot commented 3 years ago

Good point @TWiStErRob . The only androidx.test packages that have significant androidx dependencies are espresso.contrib and espresso.accessibility. So if you don't use those artifacts, you might be ok switching to androidx.test

jgavris commented 3 years ago

For those of you still having trouble with this, I think I've found a solution that at least works for our test app setup (target SDK 30 on an SDK 30 device). I put this in the test app manifests and now I'm able to communicate with the test orchestrator service.

https://developer.android.com/reference/android/Manifest.permission#QUERY_ALL_PACKAGES

krisjwolff commented 3 years ago

I am still seeing IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService following all these steps. Simply using the vanilla "new project" in Android Studio, and then adding orchestrator, I get this issue on API30 using 1.3.1-alpha02. Can someone please post the code for a project where orchestrator works using API30?

prfarlow1 commented 3 years ago

@krisjwolff Are you using an emulator? If so, I have found that when also using mocking libraries in instrumentation tests (like mockk or mockito), specifying the emulator as 64 bit makes them run. Check out my comment over on mockk: https://github.com/mockk/mockk/issues/466#issuecomment-743264258 However, I'm not sure why this would explain 1.3.1-alpha02 working on a vanilla new project in Android Studio, if that project did not include a mocking library.

krisjwolff commented 3 years ago

@krisjwolff Are you using an emulator? If so, I have found that when also using mocking libraries in instrumentation tests (like mockk or mockito), specifying the emulator as 64 bit makes them run. Check out my comment over on mockk: mockk/mockk#466 (comment) However, I'm not sure why this would explain 1.3.1-alpha02 working on a vanilla new project in Android Studio, if that project did not include a mocking library.

I am not mocking anything (like I said it's just the vanilla "new app" option in Android Studio + adding orchestrator). Same failure on a Samsung GS10, Pixel 3 physical device and Pixel 2 and 3 emulators running API30.

brettchabot commented 3 years ago

@krisjwolff Please see https://github.com/android/testing-samples/tree/main/runner/AndroidTestOrchestratorSample for a simple sample that should work on API 30

krisjwolff commented 3 years ago

@brettchabot Great, thanks! I am able to run orchestrated tests now on API30, on Pixel (physical device, and emulator)!!

But we are still having the same issue using a Samsung Galaxy s10 device (using our app, using the default "New App" in Android Studio, and also using your AndroidTestOrchestratorSampe app). Any clues about what might be happening there?

brettchabot commented 3 years ago

Not sure why samsung would be any different. Are there any suspicious looking errors in the logcat?

krisjwolff commented 3 years ago

@brettchabot Nothing I can see that's any different. It's basically the same exception I was seeing before updating to alpha02...

2021-02-12 11:30:00.766 19553-19553/? E/AndroidRuntime: FATAL EXCEPTION: main Process: com.example.android.testing.androidtestorchestratorsample, PID: 19553 java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService at android.app.ActivityThread.handleMakeApplication(ActivityThread.java:7427) at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7374) at android.app.ActivityThread.access$1500(ActivityThread.java:301) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2118) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:246) at android.app.ActivityThread.main(ActivityThread.java:8425) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:596) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1130) Caused by: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService at androidx.test.internal.events.client.TestEventServiceConnectionBase.connect(TestEventServiceConnectionBase.java:94) at androidx.test.internal.events.client.TestEventClient.connect(TestEventClient.java:125) at androidx.test.runner.AndroidJUnitRunner.isOrchestratorServiceProvided(AndroidJUnitRunner.java:343) at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:315) at android.app.ActivityThread.handleMakeApplication(ActivityThread.java:7422) at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7374) at android.app.ActivityThread.access$1500(ActivityThread.java:301) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2118) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:246) at android.app.ActivityThread.main(ActivityThread.java:8425) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:596) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1130) 2021-02-12 11:30:00.766 19553-19553/? E/AndroidJUnitRunner: An unhandled exception was thrown by the app. 2021-02-12 11:30:00.767 19553-19553/? E/InstrumentationResultPrinter: Failed to mark test No Tests as finished after process crash 2021-02-12 11:30:00.767 19553-19553/? E/MonitoringInstr: Exception encountered by: Thread[main,5,main]. Dumping thread state to outputs and pining for the fjords. java.lang.RuntimeException: Exception thrown in onCreate() of ComponentInfo{com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner}: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService at android.app.ActivityThread.handleMakeApplication(ActivityThread.java:7427) at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7374) at android.app.ActivityThread.access$1500(ActivityThread.java:301) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2118) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:246) at android.app.ActivityThread.main(ActivityThread.java:8425) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:596) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1130) Caused by: java.lang.IllegalStateException: Cannot connect to androidx.test.orchestrator.OrchestratorService at androidx.test.internal.events.client.TestEventServiceConnectionBase.connect(TestEventServiceConnectionBase.java:94) at androidx.test.internal.events.client.TestEventClient.connect(TestEventClient.java:125) at androidx.test.runner.AndroidJUnitRunner.isOrchestratorServiceProvided(AndroidJUnitRunner.java:343) at androidx.test.runner.AndroidJUnitRunner.onCreate(AndroidJUnitRunner.java:315) at android.app.ActivityThread.handleMakeApplication(ActivityThread.java:7422) at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7374) at android.app.ActivityThread.access$1500(ActivityThread.java:301) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2118) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loop(Looper.java:246) at android.app.ActivityThread.main(ActivityThread.java:8425) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:596) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1130)

KimKitaeB commented 3 years ago

it doesn't work on SAMSUNG device, It's still hang so I can't start to test even I used 1.3.1 Alpha03. what do I can something to do?

krisjwolff commented 3 years ago

@brettchabot Actually, I am seeing this exception on the Samsung device (replaced our test app's package with the word "package" -- and yes, that package IS installed on the device):

2021-02-16 10:23:03.356 19848-19874/androidx.test.orchestrator E/AndroidRuntime: FATAL EXCEPTION: AndroidTestOrchestrator
    Process: androidx.test.orchestrator, PID: 19848
    java.lang.IllegalStateException: Package [package] cannot be found on the system.
        at androidx.test.orchestrator.AndroidTestOrchestrator.getTargetPackage(AndroidTestOrchestrator.java:7)
        at androidx.test.orchestrator.AndroidTestOrchestrator.access$600(AndroidTestOrchestrator.java:1)
        at androidx.test.orchestrator.AndroidTestOrchestrator$2.run(AndroidTestOrchestrator.java:4)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

brettchabot commented 3 years ago

reopening so its on our radar, will try to repro

brettchabot commented 3 years ago

Thanks again for reporting this.

We were able to reproduce on a Samsung and Pixel 4 device. We're still confused why devices have different behavior than emulators, and are following up with the Android platform team.

In the meantime, there is a workaround.

Download the orchestrator and service apks from https://maven.google.com/web/index.html#androidx.test:orchestrator:1.4.0-alpha04 and https://maven.google.com/web/index.html#androidx.test.services:test-services:1.4.0-alpha04 respectively
Install with 'adb install -g -r --force-queryable '

Then run tests.

We're also working with the android gradle team to put this solution into AGP.

krisjwolff commented 3 years ago

@brettchabot This DOES make your sample app work on API 30 devices, thanks! Our app still does not for some reason... There is no apparent reason in the logs... I will need to dig in and see what's going on ...

Update: The only thing in the logcat on the device is the same

2021-02-26 12:46:35.812 15266-15293/androidx.test.orchestrator E/AndroidRuntime: FATAL EXCEPTION: AndroidTestOrchestrator
    Process: androidx.test.orchestrator, PID: 15266
    java.lang.IllegalStateException: Package [our package name here] cannot be found on the system.
        at androidx.test.orchestrator.AndroidTestOrchestrator.getTargetPackage(AndroidTestOrchestrator.java:7)
        at androidx.test.orchestrator.AndroidTestOrchestrator.access$600(AndroidTestOrchestrator.java:1)
        at androidx.test.orchestrator.AndroidTestOrchestrator$2.run(AndroidTestOrchestrator.java:4)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

krisjwolff commented 3 years ago

@brettchabot Ah-ha, I see why it's not working in our app... We need to clear the data from the app between tests, so we include this line in our build.gradle:

testInstrumentationRunnerArguments clearPackageData: 'true'

When I add this option to your com.example.android.testing.androidtestorchestratorsample code, it also fails in the same way on physical devices. Any ideas how to solve?

cmcheungapp commented 3 years ago

I am also using testInstrumentationRunnerArguments clearPackageData: 'true' Just update core and runner versions to make it working on android 11:

androidTestImplementation 'androidx.test:core-ktx:1.4.0-alpha04'
androidTestImplementation 'androidx.test:runner:1.4.0-alpha04'
androidTestUtil 'androidx.test:orchestrator::1.4.0-alpha04'

Treat it as a reference.

krisjwolff commented 3 years ago

@cmcheungapp I have attempted what you suggested. I get the same result on Samsung G10 and Pixel 5 devices. Didn't help.

brettchabot commented 3 years ago

I'm seeing the same result as @krisjwolff : Using clearPackageData 'true; with com.example.android.testing.androidtestorchestratorsample indeed fails on android 11 Pixel 4), with the --force-queryable workaround.

We're investigating

krisjwolff commented 3 years ago

@brettchabot Thanks for looking! We currently have a hackish work-around (clearing the shared-prefs store before and after each test), but we'd love to no longer need to do that :)

brettchabot commented 3 years ago

Please give Studio Beta 6 and androidx.test orchestrator 1.4.0-alpha05 a try. This should fix the issues running tests on android 11+ with clearPackageData true, without requiring any workarounds.

krisjwolff commented 3 years ago

@brettchabot Updating to alpha05 fixes the clearPackageData issue, thanks!

However, it still needs to be installed with the adb install workaround mentioned in https://github.com/android/android-test/issues/743#issuecomment-781514509 Because of the required workaround, I feel like this issue should still be open. Is there any progress on fixing that particular issue?

brettchabot commented 3 years ago

are you using studio 4.2 beta 6 ?

QAutomatron commented 3 years ago

Using 1.4.0-alpha05 testx and Studio 4.2b6, getting "No tests to run", most probably something crashed during init phase on Android 11 emulator.

logcat said about some new permission issue:

2021-03-26 18:21:05.133 9240-9263/androidx.test.orchestrator E/OrchestrationXmlTestRunListener: Failed to generate report data
    java.io.IOException: Operation not permitted
        at java.io.UnixFileSystem.createFileExclusively0(Native Method)
        at java.io.UnixFileSystem.createFileExclusively(UnixFileSystem.java:317)
        at java.io.File.createTempFile(File.java:2018)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.getResultFile(OrchestrationXmlTestRunListener.java:2)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.createOutputResultStream(OrchestrationXmlTestRunListener.java:8)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.generateDocument(OrchestrationXmlTestRunListener.java:3)
        at androidx.test.orchestrator.listeners.OrchestrationXmlTestRunListener.orchestrationRunFinished(OrchestrationXmlTestRunListener.java:4)
        at androidx.test.orchestrator.AndroidTestOrchestrator.finish(AndroidTestOrchestrator.java:1)
        at androidx.test.orchestrator.AndroidTestOrchestrator.executeNextTest(AndroidTestOrchestrator.java:2)
        at androidx.test.orchestrator.AndroidTestOrchestrator.runFinished(AndroidTestOrchestrator.java:10)
        at androidx.test.orchestrator.TestRunnable.run(TestRunnable.java:13)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

I don't know what else to try.

btseytlinTCP commented 3 years ago

FYI 1.4.0-alpha06 released which has a fix for the forceQueryable bug: https://github.com/android/android-test/releases/tag/androidx-test-1.4.0-alpha06

android / android-test