Introduce support for zero invocations in test templates and parameterized tests

johnchurchill commented 6 years ago

Overview

Using: JUnit: 5.2.0

Great job on the new ParameterizedTest support in v.5. The replacement of the static, one-per-class Parameters annotation with more flexible MethodSource, etc. has been like a breath of fresh air and allowed me to remove thousands (!!) of lines of supporting code from my system. I'm really loving it. However, someone decided to disallow zero parameters with this precondition check in ParameterizedTestExtension.

.onClose(() ->
    Preconditions.condition(invocationCount.get() > 0,
    "Configuration error: You must provide at least one argument for this @ParameterizedTest"));

The problem with this is that we have some testing situations where parameterized tests with zero parameters are not exceptional. For example, we run tests against thousands of a certain type of class generically against a database of past production failures, and many of these classes have never experienced a production failure. When the tests are run, we now get failures due the above precondition check. JUnit 4 handled this cleanly: it would simply not run any tests on those classes.

Workaround

I can get around this by adding a null to the method creating the collection of parameters if it is empty, and then return from the beginning of the @ParameterizedTest method code if the passed parameter is null. That lets us continue to run the parameterized tests against all of the classes, but comes with some disadvantages:

We must add specific code to the front and back of every parameterized pair just to avoid a failure that doesn't matter to the tests.
The handling of the null causes the test count inflation for these "phantom" tests.
null is now reserved as a signal for no parameters, rather than something wrong in the parameter creation.

Proposal

If nobody has any strong feelings about disallowing the no-parameter case, can we just have this precondition removed from a future version?

Thanks.

sbrannen commented 6 years ago

Tentatively slated for 5.3 RC1 for team discussion

marcphilipp commented 6 years ago

@johnchurchill Could you please provide a few more details about your use case, particularly how you've written your tests? I assume it involves base classes that inherit @ParameterizedTest methods?

gaganis commented 6 years ago

I have also been trying to envision what you are trying to do and was not able.

My best guess is that you have some sort of ArgumentsProvider that returns no elements, but still I am not sure. If my guess is valid, this behaviour was explicitly added as a solution for #923.

It would be great if you could provide a skeleton test that exhibits your issue.

johnchurchill commented 6 years ago

Yes, there is a superclass of all tests which does the parameter loading and execution. There are thousands of subclasses that implement all kinds of functionality. Each (non-test) class is the logic behind an interactive web page in the system. Here is a boiled-down version of that superclass:

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SpringDevTestServerConfiguration.class)
@TestExecutionListeners(listeners = { DependencyInjectionTestExecutionListener.class })
@ComponentScan(lazyInit = true)
@TestInstance(PER_CLASS)
public abstract class AbstractConfigTestClass implements ApplicationContextAware {

    protected List<SingleConfigTestDescriptor> configTests() {
        String testClassName = this.getClass().getName();
        // remove "Test" on the end to obtain the config class
        String clzName = testClassName.substring(0, testClassName.length() - 4);
        try {
            Class configClz = Class.forName(clzName);
            List<SingleConfigTestDescriptor> params = ConfigToTestParams.getTestParameters(configClz);
            if (params.size() == 0) {
                params.add(null); // need to add null here to avoid junit 5 failing
            }
            return params;
        }
        catch (ClassNotFoundException e1) {
            throw new RuntimeException("Could not find expected class: " + clzName, e1);
        }
    }

    @ParameterizedTest
    @MethodSource("configTests")
    public void runTest(SingleConfigTestDescriptor test) {
        if (test == null) {
            return; // nothing to do; this is the null to avoid the junit 5 failure
        }
        // use the test data to run the test against the class
    }

}

By the time we know that there are no available tests to run, it's too late.. JUnit 5 will throw an exception. There are 6 lines in there whose only purpose is to make it run successfully like it did under JUnit 4. I just started working with JUnit 5 a couple of days ago, so I haven't had the chance to look into using ArgumentsProvider. Would using ArgumentsProvider avoid the failure?

sbrannen commented 6 years ago

In any case, we should improve the error message.

"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".

The difference being "argument" vs. "invocation context".

And the invocation context can still supply zero "arguments" to the method, though that's not the topic of this issue.

sbrannen commented 6 years ago

FYI: I reworded the title of this issue accordingly.

sbrannen commented 6 years ago

Keep in mind that a @TestTemplate method in general cannot currently have zero invocations.

sbrannen commented 6 years ago

In other words, see org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(int). 😉

johnchurchill commented 6 years ago

I took a look at TestTemplateTestDescriptor, and yes the zero invocations would be stopped there too. I tried taking both of those preconditions out (TestTemplateTestDescriptor and ParameterizedTestExtension) and making a simple class with one @Test and one @ParameterizedTest which refers to a @MethodSource which returns an empty list. The result is that 1/1 tests pass (just bar2()), which is the same behavior we had in JUnit 4. There is no side-effect of allowing the @ParameterizedTest to run with no invocations.

public class FooTest {

    @ParameterizedTest
    @MethodSource("data")
    void bar1(String s) {
    }

    @Test
    public void bar2() {
        // pass
    }

    public static List<String> data() {
        // pretend we have no test cases in the QA database
        return new ArrayList<>();
    }
}

sbrannen commented 6 years ago

There is no side-effect of allowing the @ParameterizedTest to run with no invocations.

Well, that's only partially true.

It's a breaking change in the sense that configuration errors previously caused the parameterized test container to fail; whereas, now such configuration errors would be silently ignored.

Of course, your use case is not a "configuration error". So we have to decide how to best support both scenarios. 😉

I suppose, if we wanted to support zero invocations, we could simply generate a log message informing the user. The tricky part is what log level to choose. Is that DEBUG, INFO, or WARN?

sbrannen commented 6 years ago

The result is that 1/1 tests pass

In such cases, I don't think the parameterized test container should be marked as "successful" (or somehow ignored, whatever was the case when you implemented that PoC).

Rather, I think we should mark the parameterized test container as "skipped" and provide a "reason" why it was skipped.

@junit-team/junit-lambda, thoughts?

sbrannen commented 6 years ago

As a side note, since I'm the author of the Spring TestContext Framework...

@johnchurchill, out of curiosity, why do you disable all Spring test listeners except DependencyInjectionTestExecutionListener?

That's enabled by default.

Were other listeners somehow causing problems?

gaganis commented 6 years ago

This case reminds me of the similar case #1298 which was handled by the --fail-if-no-tests cl option.

The similarity is that they both exhibit the same core dilemma: If I have zero elements to process, is that because the user has done some error or is this lack of items actually valid and intentional? The error could be in configuration as in both cases or programmatic in the case of providing arguments(invocation contexts).

The difference between the two is the default is reversed.

In the case of the Console Launcher. In cl the default was not to fail and we added an option to change this behavior.
For the parameterized case we fail if zero invocation contexts.

Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.

For me personally I think the current default to fail if no elements are present and in esense guarding against error is important. I seen many case where something was silently turned off due to error and the error was missed. *Also adding a skipped note in the execution plan could be in many cases be effectively silent if it does not actually fail a test run

gaganis commented 6 years ago

n any case, we should improve the error message.

"Configuration error: You must provide at least one argument for this @ParameterizedTest" --> "Configuration error: You must provide at least one invocation context for this @ParameterizedTest".

The difference being "argument" vs. "invocation context".

I don't think invocation context would be more meaningful for the normal user. Though indeed it is more meaningful for someone who understands the internal of the platform.

If you consult the relevant section in the User Guide about ParameterizedTests there is no mention, let alone an explanation, of "Invocation Context".

sbrannen commented 6 years ago

Sure.... "invocation context" is very technical.

How about the following instead?

"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"

sbrannen commented 6 years ago

Good points, @gaganis.

Maybe we could take a similar approach also in the case- with some sort of flag or parameter eg on the annotation.

I tend to agree that adding an opt-in flag as a annotation attribute in @ParameterizedTest would be a better, more robust, and more user friendly solution.

How about the following?

@ParameterizedTest(failIfNoArgumentsProvided = false)

... with the default being true.

Though, failIfNoArgumentsProvided is a bit verbose. Perhaps we can come up with something more succinct.

sbrannen commented 6 years ago

Maybe the following is better:

@ParameterizedTest(requireArguments = false)

gaganis commented 6 years ago

How about the following instead?

"Configuration error: You must configure at least one set of arguments for this @ParameterizedTest"

That is exactly was I was thinking now too :)

sbrannen commented 6 years ago

@johnchurchill, would the following proposal suit your needs?

@ParameterizedTest(requireArguments = false)

sbrannen commented 6 years ago

FYI: I improved the error message in b1e533cf64bedb7ebbb8c80e9141fbb4cd515d0d.

johnchurchill commented 6 years ago

Yes, requireArguments = false would suit my needs perfectly. I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.

sbrannen commented 6 years ago

Yes, requireArguments = false would suit my needs perfectly.

Thanks for the feedback.

I hate to see more corner-case parameters being introduced though. The simple API is what makes JUnit so approachable.

Well, therein lies the rub: We can't have our cake and eat it, too.

As already pointed out, switching the behavior would no longer be user friendly regarding user configuration errors. So, IMHO, the only viable option is to make it an opt-in feature while retaining backwards compatibility and upholding UX.

Of course, if anyone has any better ideas, I'm all ears.

marcphilipp commented 6 years ago

@johnchurchill You can just use an assumption in your configTests() method like this to skip the complete @ParameterizedTest:

assumeFalse(params.isEmpty(), "nothing to test");

sbrannen commented 6 years ago

Indeed, a failed assumption in the factory method is an easy way to achieve this without changes to existing APIs.

Nice idea, @marcphilipp! 👍

marcphilipp commented 6 years ago

Actually, it was @sormuras who had the idea during yesterday’s team call. 😉

sbrannen commented 6 years ago

Aha... well then... in case that.... @sormuras 👏

johnchurchill commented 6 years ago

Hi, Thanks for the suggestion. I just got to a stopping point where I could try out the assumption approach. However, either I'm misunderstanding where it goes, or it doesn't apply to this situation. In the docs, the assumptions abort a test within a test-annotated method, not within the method that gathers and returns the test parameters. TestAbortedException is thrown, which causes an error. Here is how I am using it, modified from my original code above.

@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = SpringDevTestServerConfiguration.class)
@ComponentScan(lazyInit = true)
@TestInstance(PER_CLASS)
public abstract class AbstractConfigTestClass implements ApplicationContextAware {

    protected List<SingleConfigTestDescriptor> configTests() {
        String testClassName = this.getClass().getName();
        // remove "Test" on the end to obtain the config class
        String clzName = testClassName.substring(0, testClassName.length() - 4);
        try {
            Class configClz = Class.forName(clzName);
            List<SingleConfigTestDescriptor> params = ConfigToTestParams.getTestParameters(configClz);
            Assumptions.assumeFalse(params.isEmpty(), "nothing to test");
            return params;
        }
        catch (ClassNotFoundException e1) {
            throw new RuntimeException("Could not find expected class: " + clzName, e1);
        }
    }

    @ParameterizedTest
    @MethodSource("configTests")
    public void runTest(SingleConfigTestDescriptor test) {
        // use the test data to run the test against the class
    }

}

Result when no test parameters exist:

org.opentest4j.TestAbortedException: Assumption failed: nothing to test

sbrannen commented 6 years ago

TestAbortedException is thrown, which causes an error.

Can you be more explicit about what error you see (i.e., reported for what)?

sbrannen commented 6 years ago

Never mind. I simplified the test case as follows in order to reproduce the error locally.

import java.util.List;

import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

class TestCase {

    static List<String> strings() {
        Assumptions.assumeFalse(true, "nothing to test");
        return null;
    }

    @ParameterizedTest
    @MethodSource
    void strings(String test) {
    }

}

sbrannen commented 6 years ago

For the above TestCase, I can confirm that a failed assumption results in a failure for the strings(String) parameterized test method as follows when executing against the current master.

Note the suppressed PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest.

org.opentest4j.TestAbortedException: Assumption failed: nothing to test
    at org.junit.jupiter.api.Assumptions.throwTestAbortedException(Assumptions.java:255)
    at org.junit.jupiter.api.Assumptions.assumeFalse(Assumptions.java:191)
    at TestCase.strings(TestCase.java:10)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:515)
    at org.junit.jupiter.params.provider.MethodArgumentsProvider.lambda$1(MethodArgumentsProvider.java:46)

// omitted

    at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:101)
    at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:1)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:131)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1378)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:135)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1378)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$4(NodeTestTask.java:135)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:127)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:107)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:52)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:184)
    at org.junit.platform.launcher.core.DefaultLauncher.lambda$6(DefaultLauncher.java:152)
    at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:166)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:145)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:92)
    at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
    Suppressed: org.junit.platform.commons.util.PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest
        at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:280)
        at org.junit.jupiter.params.ParameterizedTestExtension.lambda$9(ParameterizedTestExtension.java:87)
        at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:281)
        ... 39 more

marcphilipp commented 6 years ago

Works for me:

> Task :documentation:consoleLauncherTest
╷
├─ JUnit Jupiter ✔
│  └─ AssumptionFailureInMethodSourceTests ✔
│     └─ strings(String) ■ Assumption failed: nothing to test
└─ JUnit Vintage ✔

I added a test to master that documents the current behavior: d5df07c5992440f2c93a68ebd642b036247c406b

@johnchurchill @sbrannen Where did you see the failing test? Was it in an IDE?

sbrannen commented 6 years ago

Where did you see the failing test? Was it in an IDE?

That was in Eclipse Oxygen (4.7.3a).

sbrannen commented 6 years ago

I added a test to master that documents the current behavior

Thanks, @marcphilipp

johnchurchill commented 6 years ago

A couple of things.

First, I am running Eclipse Photon 4.8 when I got the error. But if I run the test in Gradle, the assumption (within the @MethodSource-identified method) causes the test to be ignored, no error. So, it seems like the issue is more a problem for the way Eclipse is running tests, not JUnit itself. But only because of the TestAbortedException which is thrown from within the assumption.

Second, I stumbled upon the @TestFactory feature in the docs and realized that it is a better paradigm for what I was doing back in JUnit 4. Instead of creating a collection of data-driven parameter objects, I should create a collection of data-driven tests. I did a quick test and returned an empty Collection<DynamicTest>. In both Eclipse and Gradle, no tests were executed. This really seems to be a better approach and causes no issue. It will take some time for me to get around to rewriting the test framework to use it, but right now it looks like @TestFactory is a real winner of a feature.

    @TestFactory
    Collection<DynamicTest> testdyn() {
        return new ArrayList<>();
    }

nipafx commented 6 years ago

Accidentally stumbled into this very interesting issue and discussion... @johnchurchill Are you happy with dynamic tests? If they solve your problem, can this issue be closed? Is the JUnit team still considering implementing the option on @ParameterizedTest?

sbrannen commented 6 years ago

Is the JUnit team still considering implementing the option on @ParameterizedTest?

This issue was assigned to the General Backlog to see if there is (eventually) significant interest from the community before outright closing it.

johnchurchill commented 6 years ago

Yes, I am happy with the dynamic tests, although Eclipse doesn't really display them properly (it undercounts the total, so I get something like Runs: 392/23, and the tests themselves are unrooted), but that is just cosmetic).

I don't think the @ParameterizedTest should be cluttered with another parameter. It doesn't make sense to have it when the parameter data will ordinarily be static.

Thanks for the great library. It makes testing even more addictive than it was before.

sbrannen commented 6 years ago

Yes, I am happy with the dynamic tests, although Eclipse doesn't really display them properly (it undercounts the total, so I get something like Runs: 392/23, and the tests themselves are unrooted), but that is just cosmetic).

@johnchurchill, that just means you're using an old version of Eclipse. Eclipse has first-class support for JUnit 5 since Eclipse Oxygen 4.7.1a. The latest version of Eclipse (Eclipse Photon 4.8) naturally also supports JUnit 5.

Long story, short: just upgrade to the latest Eclipse release. 😉

sbrannen commented 6 years ago

Hmmm... I just noticed that you sated:

First, I am running Eclipse Photon 4.8

Are you certain you saw unrooted entries for dynamic tests in Eclipse Photon, b/c if so that would be a regression in Eclipse.

johnchurchill commented 6 years ago

Sorry for the delay, I didn't notice the replies. Here is a screenshot of the JUnit tab in Eclipse Photon. Something I forgot to mention is that I am using a test suite and @SelectPackages with two package names. Each of the 37 test classes from those packages return 14 dynamic tests, for a total of 518. There are 37 entries under "JUnit Jupiter", and I think the 518 individual tests are all under "unrooted". I assume this has something to do with using the JUnit 4 suite support.

sbrannen commented 6 years ago

I assume this has something to do with using the JUnit 4 suite support.

Yes, exactly!

That behavior is to be expected since JUnit 4's reporting infrastructure is not 100% compatible with the JUnit Platform.

As a side note, if you annotate your test suite class with @UseTechnicalNames you should no longer have unrooted tests; however, @DisplayName will then be ignored.

pedrolamarao commented 5 years ago

I've just come to this issue after a web search. I am using parameterized tests with Lifecycle.PER_CLASS and abstract instance source methods. I use this technique in a TCK library meant to be specialized by providers of a certain API. There are cases in which a particular provider supports no parameters for a certain test. We built TCK instances with empty parameter sources as required by the Java compiler -- because source methods are abstract. Is this a supported use case? If so, I believe empty sources should cause no errors.

Adrodoc commented 5 years ago

We use Paramterized Tests defined in a super interface that test certain aspects of fields in the class unter test via reflection. Unfortunately we have to use @TestInstance(Lifecycle.PER_CLASS) in order to get a reference to the class under test. I was really surprised to see that method sources need to be static otherwise. We too have the issue that sometimes there is no field that is interesting for the test. This causes the test to fail. Something like @ParameterizedTest(requireArguments = false) would be great! Using assumeFalse(result.isEmpty()) in the method source is a sufficient workaround, but the code is a bit ugly, because i have to convert my streams into collections, because i cant ask my stream whether it is empty and then give it to JUnit.

Here is a simplified version of what i am doing:

import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeFalse;
import java.lang.reflect.Field;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.junit.jupiter.api.MethodOrderer.Alphanumeric;
import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import com.google.common.reflect.TypeToken;

@TestMethodOrder(Alphanumeric.class)
@TestInstance(Lifecycle.PER_CLASS)
public interface FieldsHaveNullabilityAnnotationTest<U> {
  default Class<U> getClassUnderTest() {
    @SuppressWarnings("serial")
    TypeToken<U> typeToken = new TypeToken<U>(getClass()) {};
    @SuppressWarnings("unchecked")
    Class<U> classUnderTest = (Class<U>) typeToken.getRawType();
    return classUnderTest;
  }

  default Field[] allFields() {
    Field[] result = getClassUnderTest().getDeclaredFields();
    assumeFalse(result.length == 0, "No fields found"); // Workaround for https://github.com/junit-team/junit5/issues/1477
    return result;
  }

  @ParameterizedTest
  @MethodSource("allFields")
  default void test(Field field) {
    assertTrue(field.isAnnotationPresent(Nullable.class) || field.isAnnotationPresent(Nonnull.class));
  }
}

I can then use this interface in all my unit tests:

public class Pojo {
  String string;
}

public class PojoTest implements FieldsHaveNullabilityAnnotationTest<Pojo> {
  // ...
}

sormuras commented 5 years ago

I'd resort to https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests here. Something like this:

class PojoTest implements FieldsHaveNullabilityAnnotationTest<Pojo> {
  @Override
  public Class<Pojo> getClassUnderTest() {
    return Pojo.class;
  }
}

class Pojo {
  String string;
}

@Retention(RetentionPolicy.RUNTIME)
@interface X {}

interface FieldsHaveNullabilityAnnotationTest<U> {
  Class<U> getClassUnderTest();

  @TestFactory
  default Stream<DynamicTest> testAllFields() {
    return Arrays.stream(getClassUnderTest().getDeclaredFields()).map( field ->
            DynamicTest.dynamicTest(field.getName(), () -> field.isAnnotationPresent(X.class))
    );
  }
}

quiram commented 4 years ago

Is the JUnit team still considering implementing the option on @ParameterizedTest?

This issue was assigned to the General Backlog to see if there is (eventually) significant interest from the community before outright closing it.

I'd like to express my interest on this. I just found this thread after having experienced the issue myself. My use case, in case it helps, is as follows:

We have a class with a number of subclasses that do certain operations (akin to a Command Pattern).
All the child classes have certain mandatory options and certain optional options.
We have an abstract base class for testing where one test goes over the mandatory options and the other over the optional options. This is done, like others have shown, using non-static @MethodSource and a PER_CLASS lifecycle.
For each of the child classes (the "commands"), there is a test class that extend the abstract base class, providing an implementation for the @MethodSource references.
For the first time today, we have a command that has no optional options, all of them are mandatory. This means the relevant method source is to return an empty set of arguments, which JUnit is not allowing.

Being able to use @ParameterizedTest(requireArguments = false) would be fantastic.

marcphilipp commented 4 years ago

@quiram In the meantime, how about implementing a custom ExecutionCondition or using @EnabledIf (introduced in 5.7 M1)?

quiram commented 4 years ago

Thanks @marcphilipp. Those are good suggestions, but maybe slightly heavy-handed for my case (or at least I cannot think of a simple way to leverage them). For now I'll use the assumeFalse() workaround mentioned above and I'll keep an eye on the evolution of this thread.

Thanks for your effort guys 🙂

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

stellingsimon commented 3 years ago

@stalebot, this is still an issue I'd like to see a solution for.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

junit-team / junit5