Requesting support similar to AndroidJUnitRunner CustomFilter filtering

[kartik1o]

Is your feature request related to a problem? Please describe. CustomFilter provides the ability to filter test run-time based on the characteristics of the device under test.

One use case we have -> We have a device pool consisting of tablet and phone devices. we want a test to be only executed on phone devices and similarly for tablet devices.

@Retention(value = AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.CLASS, AnnotationTarget.FUNCTION)
@CustomFilter(filterClass = TestDeviceFilter::class)
// TestDeviceFilter class contains logic to determine the device type and will evaluate if test should run or skipped 
annotation class DeviceType(val deviceType: TestDeviceType)

@RunWith(AndroidJUnit4::class)
Class DeviceTypeTest {

@Test
@DeviceType(TestDeviceType.TABLET)
fun testFunctionOnTablet() {
}

@Test
@DeviceType(TestDeviceType.PHONE)
fun testFunctionOnPhone() {
}

}

testFunctionOnTablet() should run on a tablet device and testFunctionOnPhone() should run on a phone device with single gradle invocation. ./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.sampleapp.espresso.DeviceTypeTest -Pandroid.testInstrumentationRunnerArguments.filter=com.sampleapp.espresso.filter.TestDeviceFilter

Describe the solution you'd like It would be nice if Marathon supports this filtering as well. or, is there any marathon filtering I can use to achieve this desired result?

Describe alternatives you've considered Is there a way to achieve this with a custom RunListener class?

Additional context While trying to test this feature with Marathon, I noticed that Test Runner reported the test in the started phase but ended up with no test status being reported, and because of this marathon considered the test as an INCOMPLETE Test.

2024-07-10 13:06:54.972 15056-15083 TestRunner com.sampleapp.app I run started: 1 tests 
2024-07-10 13:06:54.972 15056-15083 TestRunner com.sampleapp.app I run finished: 0 tests, 0 failed, 0 ignored

[Malinskiy]

Ignored tests on Android historically didn't return any status so I believe from what you're saying this is still the case. For example tests containing ignore annotation have a special handling in vendor android implementation for this reason.

I understand your use case, but the only thing I can suggest here is to separate the test runs and filter the tests by annotation. This is the cleanest approach here.

Merging these into one test run will be problematic to support and doesn't bring any performance or stability improvements, it just brings unnecessary complexity.

Implementation of such a feature would require filtering configuration per pool or some filtering that is always applied to all the test runs which is intrusive to users.

If you can propose a solution that doesn't involve UX degradation and is also simple to support considering the limitations of android - I'm happy to discuss.

But in general all marathon test runs are homogeneous, so diverging from this is strange.

Another way to achieve what you want would be a custom junit4 rule that throws assumption failure instead of being ignored by the framework. This way status will be reported back always and there are no changes necessary on the test runner side