android / testing-samples

A collection of samples demonstrating different frameworks and techniques for automated testing
Apache License 2.0
9.21k stars 3.62k forks source link

Cannot run tests with Android Test Orchestrator using `adb` #477

Closed bartekpacia closed 1 year ago

bartekpacia commented 1 year ago

Hi!

I'm trying to run tests of the AndroidTestOrchestratorSample app using this guide (specifically: this section of it). Unfortunately, it doesn't work: all I get is Aborted on am instruments stdout, and a very cryptic native stack trace in logcat.

Here's the shell script I have currently:

orchestrate.bash ```bash #!/usr/bin/env bash set -euo pipefail # This script runs native JUnit tests with the Android Test Orchestrator. # Adapted from: # * https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#enable-command # * https://stackoverflow.com/questions/63732977/how-to-use-android-test-orchestrator-in-command-line VERSION="1.5.0-alpha01" ORCHESTRATOR_APK_PATH="$HOME/Desktop/orchestrator.apk" TEST_SERVICES_APK_PATH="$HOME/Desktop/test-services.apk" if [ ! -f "$ORCHESTRATOR_APK_PATH" ]; then curl -o "$ORCHESTRATOR_APK_PATH" "https://dl.google.com/dl/android/maven2/androidx/test/orchestrator/$VERSION/orchestrator-$VERSION.apk" 2>/dev/null fi if [ ! -f "$TEST_SERVICES_APK_PATH" ]; then curl -o "$TEST_SERVICES_APK_PATH" "https://dl.google.com/dl/android/maven2/androidx/test/services/test-services/$VERSION/test-services-$VERSION.apk" 2>/dev/null fi DEVICE_API_LEVEL="$(adb shell getprop ro.build.version.sdk)" FORCE_QUERYABLE_OPTION="" if [[ "$DEVICE_API_LEVEL" -ge 30 ]]; then FORCE_QUERYABLE_OPTION="--force-queryable" fi # uninstall old versions adb uninstall androidx.test.services || true adb uninstall androidx.test.orchestrator || true # Install the test orchestrator. adb install "$FORCE_QUERYABLE_OPTION" -r "$ORCHESTRATOR_APK_PATH" # Install test services. adb install "$FORCE_QUERYABLE_OPTION" -r "$TEST_SERVICES_APK_PATH" # app_process spawns a new process # Add "-e clearPackageData true" to clear your app's data in between runs. adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process androidx.test.services.shellexecutor.ShellMain / \ am instrument -w \ -e targetInstrumentation com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner \ -e clearPackageData true \ androidx.test.orchestrator/.AndroidTestOrchestrator' ```

When I run it, all I get is Aborted (from the last adb invocation – adb shell 'CLASSPATH ...)

Full logs of running the script above ```console $ bash -x ./orchestrate + set -euo pipefail + VERSION=1.5.0-alpha01 + ORCHESTRATOR_APK_PATH=/Users/bartek/Desktop/orchestrator.apk + TEST_SERVICES_APK_PATH=/Users/bartek/Desktop/test-services.apk + '[' '!' -f /Users/bartek/Desktop/orchestrator.apk ']' + curl -o /Users/bartek/Desktop/orchestrator.apk https://dl.google.com/dl/android/maven2/androidx/test/orchestrator/1.5.0-alpha01/orchestrator-1.5.0-alpha01.apk + '[' '!' -f /Users/bartek/Desktop/test-services.apk ']' + curl -o /Users/bartek/Desktop/test-services.apk https://dl.google.com/dl/android/maven2/androidx/test/services/test-services/1.5.0-alpha01/test-services-1.5.0-alpha01.apk ++ adb shell getprop ro.build.version.sdk + DEVICE_API_LEVEL=33 + FORCE_QUERYABLE_OPTION= + [[ 33 -ge 30 ]] + FORCE_QUERYABLE_OPTION=--force-queryable + adb uninstall androidx.test.services Failure [DELETE_FAILED_INTERNAL_ERROR] + true + adb uninstall androidx.test.orchestrator Failure [DELETE_FAILED_INTERNAL_ERROR] + true + adb install --force-queryable -r /Users/bartek/Desktop/orchestrator.apk Performing Streamed Install Success + adb install --force-queryable -r /Users/bartek/Desktop/test-services.apk Performing Streamed Install Success + adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process androidx.test.services.shellexecutor.ShellMain / \ am instrument -w \ -e targetInstrumentation com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner \ -e clearPackageData true \ androidx.test.orchestrator/.AndroidTestOrchestrator' Aborted ```

And here's the logcat excerpt that I think might be relevant

Details ``` 05-08 21:56:14.935 30629 30629 E appproc : ERROR: could not find class '/' 05-08 21:56:14.935 30629 30629 F app_process: thread.cc:2468] No pending exception expected: java.lang.ClassNotFoundException: . 05-08 21:56:14.935 30629 30629 F app_process: thread.cc:2468] (Throwable with empty stack trace) 05-08 21:56:14.935 30629 30629 F app_process: thread.cc:2468] ... 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] Runtime aborting... 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] All threads: 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] DALVIK THREADS (5): 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] "main" prio=5 tid=1 Runnable ... ... 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] (no managed stack frames) 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] Pending exception java.lang.ClassNotFoundException: . 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] (Throwable with empty stack trace) 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:676] 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:684] No pending exception expected: java.lang.ClassNotFoundException: . 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:684] (Throwable with empty stack trace) 05-08 21:56:14.970 30629 30629 F app_process: runtime.cc:684] --------- beginning of crash 05-08 21:56:14.970 30629 30629 F libc : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 30629 (main), pid 30629 (main) ... ... 05-08 21:56:15.084 30657 30657 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 05-08 21:56:15.084 30657 30657 F DEBUG : Build fingerprint: 'google/sdk_gphone64_arm64/emu64a:13/TE1A.220922.021/9526604:userdebug/dev-keys' 05-08 21:56:15.084 30657 30657 F DEBUG : Revision: '0' 05-08 21:56:15.084 30657 30657 F DEBUG : ABI: 'arm64' 05-08 21:56:15.084 30657 30657 F DEBUG : Timestamp: 2023-05-08 21:56:14.998576074+0200 05-08 21:56:15.084 30657 30657 F DEBUG : Process uptime: 1s 05-08 21:56:15.084 30657 30657 F DEBUG : Cmdline: app_process androidx.test.services.shellexecutor.ShellMain / am instrument -w -e targetInstrumentation com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner -e clearPackageData true androidx.test.orchestrator/.AndroidTestOrchestrator 05-08 21:56:15.084 30657 30657 F DEBUG : pid: 30629, tid: 30629, name: main >>> app_process <<< 05-08 21:56:15.084 30657 30657 F DEBUG : uid: 2000 05-08 21:56:15.084 30657 30657 F DEBUG : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE) 05-08 21:56:15.084 30657 30657 F DEBUG : pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY) 05-08 21:56:15.084 30657 30657 F DEBUG : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr -------- 05-08 21:56:15.084 30657 30657 F DEBUG : Abort message: 'No pending exception expected: java.lang.ClassNotFoundException: . 05-08 21:56:15.084 30657 30657 F DEBUG : (Throwable with empty stack trace)' 05-08 21:56:15.084 30657 30657 F DEBUG : x0 0000000000000000 x1 00000000000077a5 x2 0000000000000006 x3 0000007fc93b7730 05-08 21:56:15.084 30657 30657 F DEBUG : x4 6f646277641f666d x5 6f646277641f666d x6 6f646277641f666d x7 7f7f7f7f7f7f7f7f 05-08 21:56:15.084 30657 30657 F DEBUG : x8 00000000000000f0 x9 00000078f1c22a00 x10 0000000000000001 x11 00000078f1c60de4 05-08 21:56:15.084 30657 30657 F DEBUG : x12 0000000000002bba x13 0000000000000097 x14 0000007fc93b6550 x15 0000050c486bc778 05-08 21:56:15.084 30657 30657 F DEBUG : x16 00000078f1cc5d58 x17 00000078f1ca2c70 x18 0000007910ae4000 x19 00000000000077a5 05-08 21:56:15.084 30657 30657 F DEBUG : x20 00000000000077a5 x21 00000000ffffffff x22 000000765e215000 x23 0000000000000002 05-08 21:56:15.084 30657 30657 F DEBUG : x24 000000790fdd3000 x25 0000007fc93b7b1c x26 0000007fc93b7b58 x27 0000007fc93b7b58 05-08 21:56:15.084 30657 30657 F DEBUG : x28 000000790fdd3000 x29 0000007fc93b77b0 05-08 21:56:15.084 30657 30657 F DEBUG : lr 00000078f1c52968 sp 0000007fc93b7710 pc 00000078f1c52994 pst 0000000000001000 05-08 21:56:15.084 30657 30657 F DEBUG : backtrace: 05-08 21:56:15.084 30657 30657 F DEBUG : #00 pc 0000000000051994 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 01331f74b0bb2cb958bdc15282b8ec7b) 05-08 21:56:15.084 30657 30657 F DEBUG : #01 pc 00000000006d2a4c /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+704) (BuildId: e24a1818231cfb1649cb83a5d2869598) 05-08 21:56:15.084 30657 30657 F DEBUG : #02 pc 0000000000016ea8 /apex/com.android.art/lib64/libbase.so (android::base::SetAborter(std::__1::function&&)::$_3::__invoke(char const*)+80) (BuildId: 420d56eac27a210c92900f3ddb760c86) 05-08 21:56:15.084 30657 30657 F DEBUG : #03 pc 0000000000016450 /apex/com.android.art/lib64/libbase.so (android::base::LogMessage::~LogMessage()+352) (BuildId: 420d56eac27a210c92900f3ddb760c86) 05-08 21:56:15.084 30657 30657 F DEBUG : #04 pc 000000000038d66c /apex/com.android.art/lib64/libart.so (art::Thread::AssertNoPendingException() const+488) (BuildId: e24a1818231cfb1649cb83a5d2869598) 05-08 21:56:15.084 30657 30657 F DEBUG : #05 pc 000000000038a398 /apex/com.android.art/lib64/libart.so (art::ClassLinker::FindClass(art::Thread*, char const*, art::Handle)+64) (BuildId: e24a1818231cfb1649cb83a5d2869598) 05-08 21:56:15.084 30657 30657 F DEBUG : #06 pc 00000000005d0b68 /apex/com.android.art/lib64/libart.so (art::JNI::FindClass(_JNIEnv*, char const*)+672) (BuildId: e24a1818231cfb1649cb83a5d2869598) 05-08 21:56:15.084 30657 30657 F DEBUG : #07 pc 00000000000c4438 /system/lib64/libandroid_runtime.so (jniRegisterNativeMethods(_JNIEnv*, char const*, JNINativeMethod const*, int)+68) (BuildId: a31474ac581b716d4588f8c97eb06009) 05-08 21:56:15.084 30657 30657 F DEBUG : #08 pc 00000000000c43a0 /system/lib64/libandroid_runtime.so (android::register_com_android_internal_os_RuntimeInit(_JNIEnv*)+104) (BuildId: a31474ac581b716d4588f8c97eb06009) 05-08 21:56:15.084 30657 30657 F DEBUG : #09 pc 00000000000c8760 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::startReg(_JNIEnv*)+64) (BuildId: a31474ac581b716d4588f8c97eb06009) 05-08 21:56:15.084 30657 30657 F DEBUG : #10 pc 00000000000c8470 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vector const&, bool)+500) (BuildId: a31474ac581b716d4588f8c97eb06009) 05-08 21:56:15.084 30657 30657 F DEBUG : #11 pc 0000000000002554 /system/bin/app_process64 (main+1280) (BuildId: a1ab812b262121cb66f7cbe228dc9674) 05-08 21:56:15.084 30657 30657 F DEBUG : #12 pc 000000000004a1f4 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+96) (BuildId: 01331f74b0bb2cb958bdc15282b8ec7b ```

I'd appreciate any help getting this script to work.

bartekpacia commented 1 year ago

When I remove that seemingly rogue / from the last adb invocation in the script, I get:

05-08 22:05:39.785 30804 30804 E appproc : ERROR: could not find class 'am'
05-08 22:05:39.785 30804 30804 F app_process: thread.cc:2468] No pending exception expected: java.lang.ClassNotFoundException: am
05-08 22:05:39.785 30804 30804 F app_process: thread.cc:2468] (Throwable with empty stack trace)
05-08 22:05:39.785 30804 30804 F app_process: thread.cc:2468]
modified last part of the orchestrate.bash script ```bash adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process androidx.test.services.shellexecutor.ShellMain \ am instrument -w \ -e targetInstrumentation com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner \ -e clearPackageData true \ androidx.test.orchestrator/.AndroidTestOrchestrator' ```
bartekpacia commented 1 year ago

Ugh, I got the order wrong.

Good: app_process / androidx.test.services.shellexecutor.ShellMain Bad: app_process androidx.test.services.shellexecutor.ShellMain /

This is the working version:

orchestrate.bash - working ```bash #!/usr/bin/env bash set -euo pipefail # This script runs native JUnit tests with the Android Test Orchestrator. # Adapted from: # * https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#enable-command # * https://stackoverflow.com/questions/63732977/how-to-use-android-test-orchestrator-in-command-line VERSION="1.5.0-alpha01" ORCHESTRATOR_APK_PATH="$HOME/Desktop/orchestrator.apk" TEST_SERVICES_APK_PATH="$HOME/Desktop/test-services.apk" if [ ! -f "$ORCHESTRATOR_APK_PATH" ]; then curl -o "$ORCHESTRATOR_APK_PATH" "https://dl.google.com/dl/android/maven2/androidx/test/orchestrator/$VERSION/orchestrator-$VERSION.apk" 2>/dev/null fi if [ ! -f "$TEST_SERVICES_APK_PATH" ]; then curl -o "$TEST_SERVICES_APK_PATH" "https://dl.google.com/dl/android/maven2/androidx/test/services/test-services/$VERSION/test-services-$VERSION.apk" 2>/dev/null fi DEVICE_API_LEVEL="$(adb shell getprop ro.build.version.sdk)" FORCE_QUERYABLE_OPTION="" if [[ "$DEVICE_API_LEVEL" -ge 30 ]]; then FORCE_QUERYABLE_OPTION="--force-queryable" fi # uninstall old versions adb uninstall androidx.test.services || true adb uninstall androidx.test.orchestrator || true # Install the test orchestrator. adb install "$FORCE_QUERYABLE_OPTION" -r "$ORCHESTRATOR_APK_PATH" # Install test services. adb install "$FORCE_QUERYABLE_OPTION" -r "$TEST_SERVICES_APK_PATH" # app_process spawns a new process # Add "-e clearPackageData true" to clear your app's data in between runs. adb shell 'CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain \ am instrument -w \ -e targetInstrumentation com.example.android.testing.androidtestorchestratorsample.test/androidx.test.runner.AndroidJUnitRunner \ -e clearPackageData true \ androidx.test.orchestrator/.AndroidTestOrchestrator' ```

I realized this after I got curious about SpeakEasy, and found its README.