Have you considered a standard for the invocation and reporting in Java as well?

[PuppyPi]

Would anyone be interested in the idea of a way of elegantly separating the part of a testing framework that an IDE has to have built in (possibly as a plugin) from the majority of it, so that the testing framework itself could be included as "just another dependency" or "utility classes" (eg, a Referenced Library in Eclipse not a Built-In Library)?

Then you could use JUnit or TestNG or something else or even a newer version of either than your IDE supports! or even something you wrote yourself that's another source-code project in your (say, Eclipse) workspace! It always struck me as odd that JUnit was basically hardcoded into Eclipse, when it seems so arbitrary (maybe it's less so with JUnit 5, I dunno I haven't dug into it, but still, what about a framework that made each .csv file in a directory or row in a csv file, or List<MyTestDatum> be reported as a separate "test case" (with its own pass/fail/skip/err result!) and the class you Run Test on just has a method that accepts one piece of data as its argument and maybe throws AssertionError (or AssertionFailedError haha). There's an infinite number of possible testing frameworks that could be made! And yet which wouldn't require any different reporting interface to the person running the tests.)

So the "universal system" proposed here could be as simple as just having an annotation like org.opentest4j.running.TestAnnotation or something, which was attached to org.junit.jupiter.api.Test or equivalent and it contains a Class reference to another class in JUnit (or TestNG or etc.) which maybe has to have a public no-args constructor and implements an interface like org.opentest4j.running.TestingFrameworkRunner or something which has a method that Eclipse passes the java.lang.Class object corresponding to whatever was open in the editor when Run Test was pressed (or java.lang.Package or a List<Class> or something if a package or folder or project was selected), and then a few interfaces and classes for reporting the basics of the progress and results like would show up in an IDE or command-line tool (names, stacktraces if available/meaningful, java-class-to-jump-to-if-double-clicked if available/meaningful, static-vs-dynamically-generated tests/suites, passed/failed/testerror/skipped, etc.).

(There could also be a very similar org.opentest4j.running.TestSuperclass thing for things like junit.framework.TestCase that identify the class as a testwise-invokable class by its superclasses/superinterfaces rather than the presence of an annotation on it or any members, but otherwise it'd be the same—just a vehicle for a reference to the TestingFrameworkRunner implementation on the project's classpath for the IDE to delegate responsibility to.)

This interface between the IDE and Testing Framework might need to be updated, but it could gracefully be binary-compatible with older versions, so that if the IDE/etc. is older than the testing framework, it just won't call the new methods or read the new fields or cast to the new subclasses of TestResult or whatever but still display quite useful results to the user (just less fancy and precise), and if the testing framework is older than the IDE, then it won't implement the new interfaces (eg, TestingFrameworkRunner3) and when the IDE checks "instanceof" it'll recognize that and not cast it. Or if new interfaces for each version is too cumbersome for the API design, it could add methods to the same interfaces like normal and IDEs/etc. could just be understood as part of the API to always use reflection to invoke methods in TestingFrameworkRunner, or something else tricky with classloading.

Either way, unless they explicitly tap into it, it's all completely and utterly invisible to the actual test writers; their JUnit/TestNG test cases will be unaltered and still be recognizable by IDEs that directly invoke JUnit/TestNG/etc. But IDE's/etc. that use the new way will be compatible with an infinite number of possible testing frameworks and have less heavy dependencies (just this lightweight API full of interfaces, not all of JUnit/etc.), and won't need a separate plugin for each testing framework anymore! (although they might want to grandfather in adapter libraries for JUnit 3/4 and older TestNG and 'pretend' that anything named "org.junit.Test" or etc. is tagged with the @TestAnnotation annotation accordingly)

It's just an idea I've had for a while; feel free to take it or leave it. But if it would ever happen, it seems like this would be the place, so I figured I'd post it here. (And sorry if this repo only likes bugs, not ideas/suggestions, in the Issues box, but I didn't notice anything in CONTRIBUTING.md to say otherwise)

(Side note: this all is actually what I assumed OpenTest was for the longest time haha, and was surprised just now when I actually checked and saw it was mostly just a few exceptions (albeit sorely needed ones!). (though I didn't notice a BugEncounteredInTestItselfNotTestedCode type one, unless that's what TestAbortedException is for) )

[marcphilipp]

What you're describing sounds a lot like what the JUnit Platform abstraction does. IDEs only have to know about the platform to execute tests. And there's even an annotation to help IDEs understand which classes are tests.

[PuppyPi]

That does seem similar.

But if @Testable has no members (like a java.lang.Class annotation-method), how can the IDE tell which TestEngine implementation to use? Assuming they can even be on the classpath of the project (in client code not IDE/plugin code)?

[marcphilipp]

The IDE doesn't have to know which TestEngine implementation to use. It only needs to pass the test class via a ClassSelector to the JUnit Platform Launcher which in turn will ask all registered engines to run the class, if they can.

[PuppyPi]

"Registered engines" there you go; I thought that's how it must work.

That's my point: how does the IDE know which engines to register when they're in a .jar (or not) included in the project's classpath aka "Referenced Libraries" for Eclipse ?

[kcooney]

how does the IDE know which engines to register

They register themselves. https://junit.org/junit5/docs/current/user-guide/#test-engines-registration

[PuppyPi]

Ahhh that; I see, thanks!

But it does perform that for Referenced Libraries on the project's classpath that are part of the project not the IDE? Ie, does it run the java.util.ServiceLoader method[s] in a ClassLoader context that includes non-IDE, project jars?

I guess that would be IDE dependent but is that the expected behavior?

[kcooney]

But it does perform that for Referenced Libraries on the project's classpath that are part of the project not the IDE?

I would expect most IDEs to spawn a new JVM to run the tests (or, possibly, run in the same VM with a fancy class loader that uses the project jars). So the service loader would see project based jars and not the jars the IDE itself uses. How the run-time class path would be constructed would depend on how the projects are configured in the IDE.

If you have more questions, I suggest asking on a JUnit 5 forum/bug. It might help to describe what problem you are trying to solve.

[PuppyPi]

Okay that's what I'd expect. JUnit Platform (as opposed to JUnit Framework) does seem to serve the purpose my original post was about! Thanks, all!