Feature request: Expose `PluggableTestRunner`

[NaromKuoch]

We are currently using TestParameterInjector for our snapshot tests. There are some UI configurations that we care for every snapshot tests such as

• light/dark mode

• screen sizes etc.

  It would be awesome if we could implement our own PluggableTestRunner to automatically inject those UI configs for those tests.

  Right now, we use @TestParameter/ @TestParameters for light/dark mode, but it is repetitive. Are there any concerns for exposing the class so we could implement our own test runner while preserving the TestParameterInjector behavior?

[nymanjens]

Hi,

While I definitely agree exposing PluggableTestRunner would have user benefit, I'm reluctant to do so at the moment, because it makes it so much harder to evolve it in the future.

In the past years, I've done multipe non-backwards-compatible changes to PluggableTestRunner, which would not have been possible if it had been public.

So unfortunately, I'd like to hold off on exposing until it has stayed constant for long enough that we can declare it 'final'.

For your specific use case though, I'm not sure using PluggableTestRunner would even be a good idea. If you want to combine lots of parameters in many tests, you could implement a TestParameterValuesProvider that returns all possible combinations of these parameters and use it to inject a single parameter in your tests. But even just defining an enum for all of them IMO is quite elegant:

class MyTest {
  @TestParameter LightDarkMode lightDarkMode;
  @TestParameter ScreenSize screenSize;
  ... 
}

Sure, you'd have to repeat N lines of code for each option, but it is documenting to the reader that these are in fact all considered in the test. And if N becomes too large, your tests are also going to become slower and slower due to the combinatorial explosion.

[NaromKuoch]

@nymanjens Thanks for your response.

In the past years, I've done multipe non-backwards-compatible changes to PluggableTestRunner, which would not have been possible if it had been public. So unfortunately, I'd like to hold off on exposing until it has stayed constant for long enough that we can declare it 'final'.

I understand your point here.

I also considered TestParameterValuesProvider option as well. Having a custom test runner (injecting a combination of configs), I was hoping to mitigate/reduce chances of missing test coverages. I know that devs still have to remember to use the specific test runner, but it'd be just simpler comparing to use TestParameterValuesProvider.

Anyways, I totally see your concerns, so we can close this issue.