search

MarathonLabs / marathon

Cross-platform test runner
https://docs.marathonlabs.io
GNU General Public License v2.0
564 stars 121 forks source link

Test bundle not reporting test annotations #948

Open DavidDTA opened 1 week ago

DavidDTA commented 1 week ago

I am running into an issue where the remote test parser is not correctly identifying test annotations. I have been following the instructions here, and I think I've done it correctly:

However, when running via the command line, I see the following in the output:

Bundle //app/build/outputs/apk/androidTest/play/debug/app-play-debug-androidTest.apk did not report any test annotations. If you need test annotations retrieval, remote test parser requires additional setup see https://docs.marathonlabs.io/runner/android/configure#test-parser

I also noticed that the output does report when it is running the am instrument command to actually run the tests, but it does not ever claim to run the am instrument -e log true command (I'm not sure whether it would when things are working properly).

Malinskiy commented 1 week ago

@DavidDTA thanks for submitting this. Could you please share Marathonfiles for cli and the one generated by gradle? Thanks

Malinskiy commented 1 week ago

From the output you've supplied above I'm not seeing the tests properly reported: the class= and test= usually indicate the test itself, i.e.

INSTRUMENTATION_STATUS: test=testAlwaysFailing
INSTRUMENTATION_STATUS: class=com.example.AbstractFailingTest

If this is working for you via gradle plugin and doesn't work in CLI then the only difference is the Marathonfile, so I'd like to check those first

DavidDTA commented 3 days ago

Here they are. They've been slightly redacted, with redacted info between <>

Here is the marathonfile generated from gradle:

---
name: <NAME>
outputDir: "/Users/<USERNAME>/.gradle/daemon/8.7/./marathon/playDebugAndroidTest"
outputConfiguration:
  maxPath: 1024
  maxFilename: 255
analyticsConfiguration:
  type: "disabled"
poolingStrategy:
  type: "omni"
shardingStrategy:
  type: "parallel"
sortingStrategy:
  type: "no-sorting"
batchingStrategy:
  type: "fixed-size"
  size: 30
  lastMileLength: 0
flakinessStrategy:
  type: "ignore"
retryStrategy:
  type: "fixed-quota"
  totalAllowedRetryQuota: 0
  retryPerTestQuota: 0
filteringConfiguration:
  allowlist:
  - type: "fully-qualified-class-name"
    regex: ".*"
  blocklist:
  - type: "annotation"
    regex: "androidx.test.filters.FlakyTest"
ignoreFailures: false
executionStrategy:
  mode: "ANY_SUCCESS"
  fast: true
isCodeCoverageEnabled: false
uncompletedTestRetryQuota: 1
includeSerialRegexes:
- ".*"
excludeSerialRegexes: []
testBatchTimeoutMillis: 1800000
testOutputTimeoutMillis: 300000
debug: true
screenRecordingPolicy: "ON_FAILURE"
vendorConfiguration:
  type: "Android"
  androidSdk: "/Users/<USERNAME>/Library/Android/sdk"
  outputs:
  - application: "<REPOSITORY>/app/build/outputs/apk/play/debug/<APP_FILENAME>.apk"
    testApplication: "<REPOSITORY>/app/build/outputs/apk/androidTest/play/debug/app-play-debug-androidTest.apk"
    extraApplications: []
  autoGrantPermission: true
  instrumentationArgs:
    clearPackageData: "true"
    <CUSTOM_ARG_1>: <CUSTOM_VALUE_1>
    <CUSTOM_ARG_2>: <CUSTOM_VALUE_2>
    <CUSTOM_ARG_3>: <CUSTOM_VALUE_3>
  applicationPmClear: false
  testApplicationPmClear: false
  adbInitTimeoutMillis: 30000
  installOptions: ""
  serialStrategy: "AUTOMATIC"
  screenRecordConfiguration:
    videoConfiguration:
      enabled: true
      width: 720
      height: 1280
      bitrateMbps: 1
      timeLimit: 180
      timeLimitUnits: "SECONDS"
    screenshotConfiguration:
      enabled: true
      width: 720
      height: 1280
      delayMs: 500
  waitForDevicesTimeoutMillis: 30000
  allureConfiguration:
    enabled: false
    relativeResultsDirectory: "/files/allure-results"
    pathRoot: "APP_DATA"
  timeoutConfiguration:
    shell: "PT20S"
    listFiles: "PT20S"
    pushFile: "PT1M"
    pushFolder: "PT1M"
    pullFile: "PT30S"
    uninstall: "PT20S"
    install: "PT1M"
    screenrecorder: "PT3M20S"
    screencapturer: "PT0.3S"
    socketIdleTimeout: "PT30S"
    portForward: "PT20S"
    boot: "PT30S"
  fileSyncConfiguration:
    pull: []
    push: []
  threadingConfiguration:
    bootWaitingThreads: 4
    adbIoThreads: 4
    installThreads: 8
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
  testAccessConfiguration:
    adb: false
    grpc: false
    console: false
    consoleToken: ""
  adbServers:
  - host: "127.0.0.1"
    port: 5037
  disableWindowAnimation: true
analyticsTracking: false
bugsnagReporting: false
deviceInitializationTimeoutMillis: 180000

Here is the Marathon file and the provided environment variables:

CUSTOM_VALUE_1="<CUSTOM_VALUE_1>"
ADB_HOST="host.docker.internal"
ADB_DEVICE_REGEX=".*"
APPLICATION_AND_TEST_APK_PATHS="[{ application: ../../app/build/outputs/apk/play/debug/<APP_FILENAME>.apk, testApplication: ../../app/build/outputs/apk/androidTest/play/debug/app-play-debug-androidTest.apk }]"
FILTERING_CONFIGURATION="{allowlist: [{type: 'fully-qualified-class-name', regex: '.*'}], blocklist: [{ type: annotation, values: [androidx.test.filters.FlakyTest]}]}"
name: <NAME>
outputDir: ../../build/outputs/marathon
screenRecordingPolicy: "ON_ANY"
vendorConfiguration:
  type: "Android"
  outputs: ${APPLICATION_AND_TEST_APK_PATHS}
  autoGrantPermission: true
  adbServers:
    - host: ${ADB_HOST}
      port: 5037
  instrumentationArgs:
    clearPackageData: true
    <CUSTOM_ARG_1>: ${CUSTOM_VALUE_1}
    <CUSTOM_ARG_2>: <CUSTOM_VALUE_2>
    <CUSTOM_ARG_3>: <CUSTOM_VALUE_3>
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
    videoConfiguration:
      enabled: false
      width: 320
      height: 480
      bitrateMbps: 1
      timeLimit: 150
    screenshotConfiguration:
      enabled: true
      width: 320
      height: 480
      delayMs: 2
  fileSyncConfiguration:
    pull:
      - relativePath: "/googletest/test_outputfiles"
        aggregationMode: DEVICE_AND_POOL
        pathRoot: EXTERNAL_STORAGE
filteringConfiguration: ${FILTERING_CONFIGURATION}
testOutputTimeoutMillis: 300000
uncompletedTestRetryQuota: 1
executionStrategy:
  mode: ANY_SUCCESS
  fast: true
batchingStrategy:
  type: "fixed-size"
  size: 5000
  durationMillis: 600000
  percentile: 80.0
  timeLimit: "-PT1H"
  lastMileLength: 10
retryStrategy:
  type: "no-retry"
analyticsTracking: false
outputConfiguration:
  maxPath: 1024
includeSerialRegexes:
  - ${ADB_DEVICE_REGEX}
analyticsTracking: false
DavidDTA commented 3 days ago

Apologies, those are the wrong files (I was on the wrong branch). Will update them shortly.

DavidDTA commented 3 days ago

Okay, the Marathonfiles have been updated. Thanks for your patience.

Malinskiy commented 3 days ago

I'm pretty sure this is not an issue with the marathon. The test parser will not report tests if the test process crashes. For example requiring test services and not having them installed on device might cause this (this installation happens automatically via gradle). You might check this by running the parsing 'am instrument' manually and looking at the last lines there

DavidDTA commented 3 days ago

What am I looking for at the end of that output? I see:

INSTRUMENTATION_STATUS: class=<myclass>
INSTRUMENTATION_STATUS: current=105
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=105
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=<mytestname>
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer.v4=[64Lorg.junit.Test(34Lexpected=class org.junit.Test$None9Ltimeout=0), 45Ldagger.hilt.android.testing.HiltAndroidTest(), 199Lkotlin.Metadata(26LbytecodeVersion=[I@ce3a01c33Ldata1=[Ljava.lang.String;@95be42533Ldata2=[Ljava.lang.String;@54199fa11LextraInt=4812LextraString=6Lkind=126LmetadataVersion=[I@b34ebab12LpackageName=)]
INSTRUMENTATION_STATUS_CODE: 2
INSTRUMENTATION_STATUS: class=<myclass>
INSTRUMENTATION_STATUS: current=105
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=105
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=<mytestname>
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: stream=

Time: 0.208

OK (105 tests)

INSTRUMENTATION_CODE: -1

Also, I noticed that using a quote rather than preformatted text caused the redactions not to appear in the original am instrument output. The class name and test name do appear in this output.

DavidDTA commented 3 days ago

I also notice that the com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer.v4 line does contain the annotation I'm trying to filter for the relevant tests

Malinskiy commented 3 days ago

The output above doesn't have any crashes. With the same device marathon cli doesn't parse the tests? Are you using the same version between gradle and cli?

DavidDTA commented 2 days ago

I've got some more information.

I had thought we were using the same version, but it turns out that gradle was using 0.10.1 and command line was using 0.10.0. After updating to 0.10.1, the issue goes away using the environment variables listed above! However, the issue comes back when listing more than one test apk.

Malinskiy commented 2 days ago

Do these bundles work in isolation?