Open harshitAtlassian opened 9 months ago
Similar issue on my side regarding the support of the @Suite.SuiteClasses
annotation :
package com.example.test_app.bar
import com.example.test_app.BaseInstrumentedTest
import org.junit.runner.RunWith
import org.junit.runners.Suite
import com.example.test_app.bar.BarInstrumentedTest
import com.example.test_app.foo.FooInstrumentedTest
@Suite.SuiteClasses(
BarInstrumentedTest::class,
FooInstrumentedTest::class
)
@RunWith(Suite::class)
class FooBarTestSuite : BaseInstrumentedTest()
It is properly supported using an adb command adb shell am instrumentation -e "class com.example.test_app.bar.FooBarTestSuite" com.test.package/com.test.runner
as well as using Gcloud CLI directly (--test-targets "class com.example.test_app.bar.FooBarTestSuite"
).
I was able to run my test suite in both cases but it fails using Flank with the following output:
RunTests
No tests for app-debug-androidTest.apk
Saved 0 shards to /path/to/the/project/app/build/fladle/virtual/results/2024-01-10_16-41-38.544741_vfbO/android_shards.json
Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/project/path...
There are no Android tests to run.
Total run duration: 0m 4s
- Preparation: 0m 2s
> Task :app:execFlankVirtual FAILED
At first sight, it seems to be either a filter or shard calculation issue, maybe in CreateAndroidTestContext
🤷♂️:
private suspend fun List<AndroidTestContext>.setupShards(): List<AndroidTestContext> = coroutineScope {
map { testContext ->
async {
val newArgs = testContext.args
val filters = TestFilters.fromTestTargets(newArgs.testTargets, newArgs.testTargetsForShard)
when {
testContext !is InstrumentationTestContext -> testContext
newArgs.useCustomSharding -> testContext.userShards(newArgs.customSharding)
newArgs.useTestTargetsForShard -> testContext.downloadApks().calculateDummyShards(newArgs, filters)
else -> testContext.downloadApks().calculateShards(newArgs, filters)
}
}
}.awaitAll().dropEmptyInstrumentationTest()
}
The dropEmptyInstrumentationTest
is suspicious to me as well as the shard calculation. It seems that it fails to compute the shards (0 shards detectedfrom the logs) and it considers there are no tests to run.
Is there any workaround I could apply to run my test suites on Firebase Test Lab using Flank? I tried to deactivate the sharding using disableSharding.set(true)
but it was not successful either. 🙏
Thanks
EDIT: I was able to find a workaround -> Adding an empty fake test succeeds in executing the test suite.
package com.example.test_app.bar
import com.example.test_app.BaseInstrumentedTest
import org.junit.runner.RunWith
import org.junit.runners.Suite
import com.example.test_app.bar.BarInstrumentedTest
import com.example.test_app.foo.FooInstrumentedTest
@Suite.SuiteClasses(
BarInstrumentedTest::class,
FooInstrumentedTest::class
)
@RunWith(Suite::class)
class FooBarTestSuite : BaseInstrumentedTest(){
@Test
fun workaroundTestToWorkWithFlank() {
}
}
That being said, it seems that if any of the tests inside the suite is failing and I set flakyTestAttempts.set(1)
, it will rerun the full suite again instead of just the failing test... The JUnit 4 @Suite.SuiteClasses
is not well supported in the library.
Also, the sharding is not efficient either in this case as it takes longer to execute 62 tests using @Suite.SuiteClasses
(Total run duration: 18m 52s) than to execute 1599 regular tests (Total run duration: 11m 33s) 🙈.
Do you think this is something that could be improved?
Describe the bug
Gcloud takes into account the
Parameterized
annotation associated with the base class while the flank requires that theParameterized
annotation to be associated with the extended class as well.To Reproduce
Example ->
GCloud
executes tests for SampleScreenshotDarkTest where asflank
doesn't.Expected behavior
Details (please complete the following information):