search

Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab
https://firebase.community/
Apache License 2.0
664 stars 112 forks source link

Parameterized test classes do not respect test-targets filters by annotation #2480

Open steverichey opened 4 months ago

steverichey commented 4 months ago

Describe the bug

An annotated test method that exists in a class annotated with @RunWith(Parameterized::class) will not be included in a test run that is filtered with test-targets to include only test methods with that annotation. An example project is available here, where the method in the parameterized test class is not included in android_shards.json if test-targets specifies - annotation flank.shard.annotationbug.Annotation.

To Reproduce

  1. Set up an Android project with test-targets specified in the Flank config file that specifies any annotation.
  2. Mark methods in a parameterized test class with the above specified annotation.
  3. Run Flank with the --dump-shards option.
  4. Note that the parameterized annotated methods are not included in the shards file.

Expected behavior

Test methods in parameterized test classes should be included or not based on the same rules that apply to non-parameterized test classes.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

I'm not sure where Flank snapshots are posted. Happy to test if a link is provided.

Post the output of flank --version.

version: v23.10.1
revision: 36e37aa9c85f5cefc5680efb699d195ee48c2bc9

Total run duration: 0m  0s

Additional context

To make the issue clear (and to verify some other configuration issue didn't cause this problem) I've created a demo project: https://github.com/steverichey/flankshardannotationbug