search

android / android-test

An extensive framework for testing Android apps
https://android.github.io/android-test
Apache License 2.0
1.16k stars 314 forks source link

Feature Request: Allow for adding comments to AndroidJUnitRunner flat test file execution option #1548

Open AaronMT opened 2 years ago

AaronMT commented 2 years ago

Currently, we (Mozilla) have large androidTest (Espresso) test suites with a lot of annotated disabled tests for @Ignore (using org.junit.Ignore). The default @Ignore JUnit Annotation allows for a default parameter of which we make use for linking back to issue trackers (e.g, @Ignore("Blocked on X")

I've been looking at the AndroidJUnitRunner documentation for exploring alternative ways to handle manifests (e.g, maintaining a remote list of disabled tests that can be updated in one spot), alongside the execution option notTestFile for listing tests we want disabled without having to annotate.

One aspect missing from a flat test manifest file is the ability to give context to the listing such as adding a comment. Wondering if this can be considered for the future for AndroidJUnitRunner (I'm all ears on other ideas too).

Example: disabled-tests.txt

org.mozilla.fenix.ui.SettingsTests # Rewrite needed to support Compose UI
brettchabot commented 2 years ago

This sounds like a reasonable FR. However, I don't think a '#' is going to work as a delimiter, because '#' is used as the class#method separator.

How about '//' ? eg

org.mozilla.fenix.ui.SettingsTests#testSettings  // Rewrite needed to support Compose UI

The testFile parser is pretty rudimentary right now. Would you be up for contributing to solve this FR?

Also note there is an open PR to deprecate testFile in favor of testFileFromStorage. #1545