search

MarathonLabs / marathon

Cross-platform test runner
https://docs.marathonlabs.io
GNU General Public License v2.0
571 stars 121 forks source link

Marathon tries to execute "#null" test #199

Closed DSamaryan closed 5 years ago

DSamaryan commented 5 years ago

I faced a strange behaviour when marathon tried to execute "#null" test, it printed the following logs:

22% | [omni]-[EMULATOR28X0X23X0] com.android.kakao.ClustersTest#choiceTest started
10:11:39.121 [AndroidDevice - execution - emulator-5554 @coroutine#148] INFO DebugTestRunListener - testEnded EMULATOR28X0X23X0 test = com.android.kakao.ClustersTest#choiceTest
23% | [omni]-[EMULATOR28X0X23X0] com.android.kakao.ClustersTest#choiceTest ended
10:11:39.650 [AndroidDevice - execution - emulator-5554 @coroutine#148] INFO DebugTestRunListener - testRunEnded elapsedTime 30130
10:11:39.651 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - handle test results EMULATOR28X0X23X0
10:11:39.654 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - request next batch for device EMULATOR28X0X23X0
10:11:39.655 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - sending next batch for device EMULATOR28X0X23X0
10:11:39.655 [main @coroutine#7] DEBUG DevicePool[omni]_DeviceActor[EMULATOR28X0X23X0] - executeBatch EMULATOR28X0X23X0
10:11:39.656 [AndroidDevice - execution - emulator-5554 @coroutine#152] DEBUG AndroidDeviceTestRunner - tests = [com.android.kakao.ClustersTest#choiceCityCluster]
10:11:49.534 [AndroidDevice - execution - emulator-5554 @coroutine#152] INFO DebugTestRunListener - testRunStarted EMULATOR28X0X23X0
10:11:49.534 [AndroidDevice - execution - emulator-5554 @coroutine#152] INFO DebugTestRunListener - testStarted EMULATOR28X0X23X0 test = com.android.kakao.ClustersTest#null
23% | [omni]-[EMULATOR28X0X23X0] com.android.kakao.ClustersTest#null started
10:11:49.535 [AndroidDevice - execution - emulator-5554 @coroutine#152] INFO DebugTestRunListener - testFailed EMULATOR28X0X23X0 test = com.android.kakao.ClustersTest#null trace = java.lang.IllegalStateException: {"statusCode":500,"statusDescription":"Internal Server Error" ...

and then it executed a batch of other tests as usual and then the problem was repeated:

10:15:13.882 [AndroidDevice - execution - emulator-5554 @coroutine#311] DEBUG AndroidDeviceTestRunner - tests = [com.android.kakao.ClustersTest#null]
10:15:16.226 [AndroidDevice - execution - emulator-5554 @coroutine#311] INFO DebugTestRunListener - testRunStarted EMULATOR28X0X23X0
10:15:16.226 [AndroidDevice - execution - emulator-5554 @coroutine#311] WARN TestRunResultsListener - skipped = com.android.kakao.ClustersTest#null, EMULATOR28X0X23X0
10:15:16.226 [AndroidDevice - execution - emulator-5554 @coroutine#311] INFO DebugTestRunListener - testRunEnded elapsedTime 1
10:15:16.228 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - handle test results EMULATOR28X0X23X0
10:15:16.228 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - request next batch for device EMULATOR28X0X23X0
10:15:16.228 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - sending next batch for device EMULATOR28X0X23X0
10:15:16.228 [main @coroutine#7] DEBUG DevicePool[omni]_DeviceActor[EMULATOR28X0X23X0] - executeBatch EMULATOR28X0X23X0
10:15:16.229 [AndroidDevice - execution - emulator-5554 @coroutine#315] DEBUG AndroidDeviceTestRunner - tests = [com.android.kakao.ClustersTest#null]
10:15:18.503 [AndroidDevice - execution - emulator-5554 @coroutine#315] INFO DebugTestRunListener - testRunStarted EMULATOR28X0X23X0
10:15:18.503 [AndroidDevice - execution - emulator-5554 @coroutine#315] WARN TestRunResultsListener - skipped = com.android.kakao.ClustersTest#null, EMULATOR28X0X23X0
10:15:18.503 [AndroidDevice - execution - emulator-5554 @coroutine#315] INFO DebugTestRunListener - testRunEnded elapsedTime 1
10:15:18.506 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - handle test results EMULATOR28X0X23X0
10:15:18.508 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - request next batch for device EMULATOR28X0X23X0
10:15:18.508 [main @coroutine#6] DEBUG QueueActor[DevicePoolId(name=omni)] - sending next batch for device EMULATOR28X0X23X0
10:15:18.508 [main @coroutine#7] DEBUG DevicePool[omni]_DeviceActor[EMULATOR28X0X23X0] - executeBatch EMULATOR28X0X23X0
10:15:18.509 [AndroidDevice - execution - emulator-5554 @coroutine#319] DEBUG AndroidDeviceTestRunner - tests = [com.android.kakao.ClustersTest#null]
10:15:21.112 [AndroidDevice - execution - emulator-5554 @coroutine#319] INFO DebugTestRunListener - testRunStarted EMULATOR28X0X23X0
10:15:21.113 [AndroidDevice - execution - emulator-5554 @coroutine#319] WARN TestRunResultsListener - skipped = com.android.kakao.ClustersTest#null, EMULATOR28X0X23X0

That was strange because no "ClustersTest#null" test had been printed before in the list of all tests.

Env: marthon 0.4.0 config:

name: "Tests"
outputDir: "/opt/marathon/shared/output/"
debug: true
vendorConfiguration:
  type: "Android"
  applicationApk: "/opt/marathon/shared/input/application.apk"
  testApplicationApk: "/opt/marathon/shared/input/test.apk"
  autoGrantPermission: true
  instrumentationArgs:
    debug: "false"
  applicationPmClear: false
  testApplicationPmClear: false
retryStrategy:
  type: "fixed-quota"
  totalAllowedRetryQuota: 15
  retryPerTestQuota: 3
xanderblinov commented 5 years ago

It happens after crash in @beforeTest block

com.malinskiy.marathon.android.TestIdentifier.toTest() tried to parse test output and got "null" test name.

@Malinskiy @tagantroy maybe a good solution is not to retry tests with "null" name?

tagantroy commented 5 years ago

200 merged, Please try latest snapshot/develop build