Discover dynamic tests early during discovery phase

[mibutec]

As discussed in https://stackoverflow.com/questions/49401619/test-discovery-of-dynamic-tests-in-eclipse-slow it would be cool, if dynamic tests could be discovered at start time of a test. Advantages are:

• No jumping of IDEs JUnit view while more ans more tests are detected
• I can start a test-class to get a list of all known tests and than start selectivly the ones I'm interested in
• When executing in IDE I get an idea of how long execution of a test-class will take based on number of tests.

Since a TestFactory returns a tupel of display name and executable code, I think it should be possible to iterate throu the returned Stream / Iterable to get all display names and create complete view on test start of a test.

[sormuras]

@mibutec Please read and understand the initial paragraph of https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests

Dynamic tests are not static. The framework will never be able to know what happens to the stream of DynamicNodes. A stream is not a collection, i.e. it does not have a pre-defined size, it does not even store data/items/tuples...

Closing this request as invalid.

[sormuras]

Maybe you're looking for Parameterized Tests?

[mibutec]

I'm sorry if I hurt you in any kind. I didn't mean to be rough.

I understand the differnce if a test needs to be known at compile time or if it needs to be known on runtime. But I think even tests defined at runtime should be known at some point in test execution, so I see there is no problem to go throu the stream as far as it's known to provide test descriptions for IDE. Maybe I'm just lacking of fantasy what dynamic tests can be used for, but I think 99% of all streams are fully known after @TestFactory annotated method finished execution.

Maybe I'm wrong, but I would like to discuss that idea. I don't see it's that aside, that it couldn't be even discussed and my issue is immediatly closed.

[sormuras]

I'm sorry if I hurt you in any kind. I didn't mean to be rough.

No offense taken, no offense meant from my side, too.

Let's discuss the idea here. Maybe the documentation and user-guide lacks detailed information regarding the purpose of the dynamic test feature. If this is the case, we need to improve it.

[mibutec]

My use case I is:

We use a lot of JBehave in my company. JBehave is a way to describe Tests in a Behavior Driven Design way. Tests are described as Scenarios in story files. In our environment several story files are executed in one JUnit test class.

Marriage between JBehave and JUnit was hard until now. In JUnit 5 I want to say a test factory: "Take a look at those story files, create tests out of them". Then several tests are created out of that story files each named as the scenario it executes. Inside the Executable there is some code utilizing JBehave core classes to run one scenario each test. For me that's one case to use Dynamic Tests, isn't it?

That all works fine so far. But one common use case is: I need to execute just one scenario out of a big story. This tests may be quite slow, so I really just want to execute that one test. In my JUnit 5 dream world I initiate test execution for the whole test, get a list of scenarios in my IDE's view, cancel test execution for the whole test class and run just the test I need. This won't work, this IDE's view doesn't know all the scenarios on test start, but after execution.

[sormuras]

Perhaps this "picture" will help to clarify the difference between static and dynamic tests.

Static / Discovery

• A @Test annotated method is our "well-known" entry-point. Just like public static void main(String... args) is the entry-point for an application.
• A @TestFactory annotated method is yet another entry-point.

• All entry-points are discovered statically ... and your build tool/IDE may show these in a test plan view.

  After discovering all entry-points, they are executed one-by-one.

Dynamic / Execution

• When-ever a @Test methods is to be executed, it is executed.
• When-ever a @TestFactory methods is to be executed, it is executed AND the returned object contains the instructions what the actual tests look like (name and executable). Now, these dynamic tests are executed one-by-one ... and your build tool is notified "on-the-fly" about the dynamically created and executed tests.

[sormuras]

[...] so I really just want to execute that one test.

Create a dedicated @Test method to pin-point the single configuration?

[sormuras]

[...] For me that's one case to use Dynamic Tests, isn't it?

True. But when you chose a dynamic tool, you'll lose static feature.

When you chose reflection, you'll lose strong static features.

[sormuras]

Marriage between JBehave and JUnit was hard until now. In JUnit 5 I want to say a test factory: "Take a look at those story files, create tests out of them". Then several tests are created out of that story files each named as the scenario it executes. Inside the Executable there is some code utilizing JBehave core classes to run one scenario each test.

Hm, implement your own TestEngine? Here you can control what is a test and how they are executed.

Here are some starting points:

• https://github.com/junit-team/junit5/wiki/Third-party-Extensions#junit-platform-test-engines
• https://junit.org/junit5/docs/current/user-guide/#launcher-api-engines-custom
• https://github.com/junit-team/junit5-samples/tree/master/junit5-modular-world/src/main/ice.cream

And when you're done, contribute it to https://github.com/jbehave/jbehave-core/jbehave-junit-engine 🚀

[mibutec]

TestEngine sounds like a good heading. I'll take a look at this. Thanks

[mibutec]

True. But when you chose a dynamic tool, you'll lose static feature.

I don't think this needs to be true in context of our discussion.

You get a Stream of names and executables out of a test factory. After the test factory is called you may go throu the stream and find all the display names to create a test description. The executables are used later.

This will work for all of your examples in https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests even generateRandomNumberOfTests(). The only case this wouldn't work is a test factory providing some tests, then taking some time to recap the tests created so far and than while the first tests already were executed, to notice: "Ahh, I forgot one, thanks god I have a stream, so I can hand in it later".

Sorry for sarcasm. I didn't find a use case for a test factory not knowning all tests when it finishes work. Can you give an example?

For me the big difference between static and dynamic tests is: Static tests need to be defined at compile time, dynamic ones not. That's already a huge, huge plus in comparsion with older JUnit versions.

[sbrannen]

Sorry for sarcasm. I didn't find a use case for a test factory not knowning all tests when it finishes work. Can you give an example?

Data being streamed in from an external resource: database, REST server, message queue, etc.

Although it's true that a test factory should not stream dynamic nodes indefinitely, the point is that such tests are dynamic in nature.

And "dynamic" tests are not discovered until the execution phase in the JUnit Platform.

For dynamic tests, only the test factory method is discovered during the discovery phase.

Note that the discovery phase precedes the execution phase, and it is during the discovery phase that the GUI in your IDE is populated with statically discovered tests.

Dynamic tests are not known until the execution phase and therefore cannot be displayed in the IDE prior to that.

[mibutec]

Thanks for clarification. By introducing that two phases, I got the problem.

[sbrannen]

Thanks for clarification. By introducing that two phases, I got the problem.

You're welcome!

Glad to be of service. 😃

[mibutec]

The more I think about it the less I get the point with the actual definition what a Dynamic Test is.

Data being streamed in from an external resource: database, REST server, message queue, etc.

For me that's an example of a datasources for parametrized tests.

For me the difference between a parametrized test and a dynamic test is:

• Parameterized Test: I have a static defined test being known on compile time, then there comes at runtime some data as parameters from some kind of datasource to repeativly execute that test with different data.
• Dynamic Test: Even the definition of a test is not known at compile time. The whole test is defined at runtime.

As examples of Dynamic Tests I think of something like:

• A TestFactory introspecting a given testee and deciding by its interfaces which (predefined) test methods need to be called for that testee, i.e. if it contains the interface Serializable than add a test testing all fields are Serializable
• Some kind of execution engine executing some kind of DSL containg test descriptions

After working a bit with TestEngines, I get deeper into the idea of discovery phase and execution phase and I think one must destinguish what can be discovered early and what needs to wait for discovery.

Parameterized and Dynamic Tests have runtime parts necessary to do the whole discovery. But Runtime doesn't necessaryly mean slow and far, far away:

• As in the Serializable example above a Dynamic test may consist of just some in-JVM Java calls
• Parameters for a Parameterized test may come from some kind of datasource
• DSL files describing a test may be resources in classpath
• Parameters for a Parameterized test may come from resources in classpath
• Test descriptions in some DSL may come from external resource

So I think you cannot say: "Dynamic Tests are always discovered in execution phase" or "Parameterized tests are always discovered in discovery phase". But it depends on the Runtime nature of your "runtime data provider".

The Framework cannot know about the nature of your "runtime data provider", it must be told by test developer. So maybe you can think of some meta information telling the Framework when to discover tests (@TestFactory(earlyDiscovery=true)) or you might make it dependent on the used return type (Stream<DynamicTest>) vs Collection<DynamicTest>)

[mibutec]

Why do I tell all this to you?

I had the last years a hard time getting some things done in JUnit 4 because the static nature of tests there. Now you come to my rescue, but what do I see: You are too dynamic.

I'd really love to have the ability to define a test without the need doing it at compile time with no other drawbacks, just behaving like any other static test.

[sbrannen]

The Framework cannot know about the nature of your "runtime data provider", it must be told by test developer. So maybe you can think of some meta information telling the Framework when to discover tests (@TestFactory(earlyDiscovery=true)) or you might make it dependent on the used return type (Stream<DynamicTest>) vs Collection<DynamicTest>)

That unfortunately will not help.

It is literally impossible to discover dynamic tests without executing the test factory method.

The execution of anything testable (e.g., @Test, @TestFactory, @ParameterizedTest methods) requires that the entire execution phase infrastructure be up and running.

Specifically, this means that all extensions must be discovered, instantiated, registered, and executed for anything testable. JUnit also must invoke all user-defined lifecycle callback methods (e.g., @BeforeAll, @BeforeEach, etc.).

So, in summary, although I understand your desire for the framework to invoke your test factory method during the discovery phase, that isn't possible since it physically requires execution of the test factory method, and "execution" is not as simple as just "invoking" the method.

[sormuras]

So, a custom test engine might be a solution here.

[sbrannen]

To make my above comments clear, I have renamed this issue to "Discover dynamic tests early during discovery phase" in order to reflect what is actually being requested.

[mibutec]

So, a custom test engine might be a solution here.

It is. So I'm on it. For now I'm trying to find out if a TestEngine is too much for what I want to achieve, or if it's making everything much better.

In the last years we arranged doing JBehave in a JUnit way, so with this in mind I don't need anything more than some Dynamic Tests.

Having now the possiblilty creating an API from scratch to execute those test, things are getting interesting.

[sbrannen]

Having now the possiblilty creating an API from scratch to execute those test, things are getting interesting.

Cool! 🤓

Let us know how it goes...

[mibutec]

It is literally impossible to discover dynamic tests without executing the test factory method.

That's clear, and now I get the problem: @TestFactory method might depend on being initialized by some Extension.

In case of @MethodSource you solved the problem by making that method static. Could that be a solution? Instead @TestFactory(earlyDiscovery=true) the keyword static in method declaration might be a signal word to allow early discovery.

I know I suck, but Dynamic Tests for me is the killer features of JUnit5, and I really was disapointed if them failed on such trivia like late feedback for developers what tests are going to be executed.

[bechte]

@mibutec To me it sounds like you are missing an Pointcut that allows you to take a discovered TestDescriptor and do some additional (even though limited) work during the discovery phase, e.g. like adding additional TestDescriptors that are resolved outside of the test Class. Currently, we are not supporting this, unfortunately.

As I was investigating quite some time in this direction, I do have a question: Would it be sufficient for your use-case, if you could implement some kind of TestProvider that is notified about every TestDescriptor found and to be able to attach children, i.e. instances of TestDescriptor to that node?

For sure, the children would be instances of TestDescriptor themselves and they would have to perform the test execution, and the test discovery by ID themselves in order to contribute.

And here comes the crucial part: We could run into conflicts if multiple TestProviders want to operate and modify the same node. This is something we could solve by limiting the richness of this feature, but there is something else in the air. Nowadays, the Jupiter engine knows in detail which constellations of TestDescriptors are valid, i.e. which follows which and who is a parent of who. In a flexible mechanism like proposed above, this kind of guarantee is missing. This also means that the whole part of Jupiters Extensions and how they are guaranteed to be executed, will be very hard to achieve. The TestDescriptors would have much more responsibility and dependencies.

If we find a way to solve the issues raising from this, and find the time to do a huge portion of refactoring, and this refactoring does not break the whole thing, I think it might be possible to provide such a mechanism.

In the end, I think you are better on writing your own TestEngine for the time being, even though I would love to provide you with such a mechanism.

[mibutec]

@bechte Mostly of all I'm missing Test Descriptions of Dynamic Tests being directly shown in my IDE after starting a test. A mechanism described by you would make it as a workaround, but letting the framework create a Test Description to overrule it by hand afterwards sounds like a featre I won't like to use.

@sbrannen @sormuras : I created locally a prototype allowing me to discover Dynamic Tests in Discovery Phase when the TestFactory method is static. Another thing looking promising: Moving the linelistener.dynamicTestRegistered(dynamicTestDescriptor); from HierarchicalTestExecutor to some stage very early in Execution Phase, so all Test Description can be shown before execution of first test.

Mostly of all both are just a bunch of lines to change with no impact on design or any architectural design desicion. So please tell me: Why is there from the beginning so much contra about the idea to provide fast feedback to developers using Dynamic Tests? Are we really talking about technical issues, or did I swim into some internal deep water or anything else?

P.S. I'm not allowed to create a branch in junit5 project (ERROR: Permission to junit-team/junit5.git denied to mibutec.). Do I need special rights or special naming conventions on branch names?

[sormuras]

P.S. I'm not allowed to create a branch in junit5 project (ERROR: Permission to junit-team/junit5.git denied to mibutec.). Do I need special rights or special naming conventions on branch names?

You have to fork the project first. Then, you may create as many branches as you want. From there, you may file pull requests upstream against the forked project.

[sbrannen]

Reopening for further team discussion...

[bechte]

@bechte Mostly of all I'm missing Test Descriptions of Dynamic Tests being directly shown in my IDE after starting a test. A mechanism described by you would make it as a workaround, but letting the framework create a Test Description to overrule it by hand afterwards sounds like a feature I won't like to use.

You wouldn't be able to overrule that TestDescriptor but you would be allowed to attach more children to that TestDescriptor. Replacing a TestDescriptor cannot be allowed, because it might come from another extension.

[mibutec]

Created:

• 1350

• 1351

1350 is less invasive and more powerful since it discovers even non-static @TestFactory tests early. Problem for now: Tests are discovered after execution phase for that @TestFactory method started. So if there are tests running before that method discovery starts after those are executed.

1351 introduces the idea of static @TestFactory methods. Compared to 1350 there are more code changes and some methods got ugly due to more method parameters, further refactoring is necessary. It is less powerful since non-static @TestFactory methods are not discovered early. But it seems to be a clean approach, since test discovery takes places in discovery phase.

Maybe those both are a good starting point for your team discussion.

[marcphilipp]

Thanks for the PRs!

Regarding #1351: IMHO executing a @TestFactory method during discovery, even when it's static, is not going to work out because these methods may have parameters to be resolved by extensions which are not registered, yet. Moreover, calling user code during discovery might cause JUnit to seem unresponsive because there's no reporting etc.

Regarding #1350: If we decide to do this, we should only do it for Collections, not for Streams, Iterators, or Iterables.

@junit-team/junit-lambda Thoughts?

[sbrannen]

I tend to agree with you @marcphilipp, but let's discuss it further during the next team call.

[sbrannen]

Team Decision: the team does not feel that the minor, aesthetic improvement proposed in these PRs would outweigh the added complexity necessary to support such a feature.

[mibutec]

Hey, the issue was open for almost whole easter, so maybe it will come as Easteregg :laughing:

Just for my curiosity: As far I could appraise the needed changes, there are just some code movements for 1350 necessary. What is added complexity?

[sbrannen]

Any introduction of new types, methods, or method parameters as well as additional behavior added to existing behavior or changes to existing behavior that are not in line with the documented, published APIs result in added complexity beyond the status quo.

[mibutec]

So, are we talking about "complexity" or about "work"? In case of work I can support you.

Afaik timing behavior of Dynamic Tests is not part of User Guide, so no changes necessary. @API annotated classes are not changed, either.

JavaDoc, Tests, Formatting, ... would be done by me.

[sbrannen]

Please read my response.

I explicitly stated "added complexity".