Distribute TCK tests as an archive instead of source code

[KyleAure]

For Jakarta 11 I think we should consider refactoring the Faces TCK so that tests are packaged into a single JAR archive that can be distributed and run by any implementation.

As it is today the TCK distribution just contains the source code, and implementations are expected to modify the source project to run against their implementation. Whereas the reference implementation is used by default. This means that anyone who wants to create a new implementation of Jakarta Faces are going to be one step behind the reference implementation.

Instead, the TCK should just package up the tests that can be consumed and run by implementation projects. Some examples I have worked on for Jakarta EE 10/11:

• Concurrency: https://github.com/jakartaee/concurrency/tree/master/tck
• Data: https://github.com/jakartaee/data/tree/main/tck

I believe this is the format many of the other Jakarta Specs have taken when creating standalone TCKs.

FYI - @brideck @pnicolucci

[arjantijms]

For Jakarta 11 I think we should consider refactoring the Faces TCK so that tests are packaged into a single JAR archive that can be distributed and run by any implementation.

In source code, the Faces TCK should be able to be run by any implementation already really.

As it is today the TCK distribution just contains the source code, and implementations are expected to modify the source project to run against their implementation

It's essentially only the main pom where a profile can be added. The platform TCK for instance required many more files to be modified.

It was proposed before though to look into a plugin that essentially functions like the Maven Invoker plug-in, but is able to add something to its class path.

Instead, the TCK should just package up the tests that can be consumed and run by implementation projects.

It's not that trivial. The Faces TCK (just like the security and authentication tck) are a multi-module project; every test is its own module (typically a war), that can be deployed, debugged and tinkered with standalone, and is isolated from other tests.

Those four things are clearly (IMHO) an advantage over putting everything in a single test jar.

[arjantijms]

This means that anyone who wants to create a new implementation of Jakarta Faces are going to be one step behind the reference implementation.

There is a placeholder in the main pom where a custom profile can be added:

https://github.com/jakartaee/faces/blob/master/tck/pom.xml#L353

Here's on example of running the TCK from another pom file:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/faces/pom.xml

If there's no suitable profile yet, that same pom could as part of the setup / init insert a profile into the target pom using the provided placeholder.

I'm not sure if just an existing profile in a pom really sets a new Faces implementation back, but if this really bothers people we could propose to remove the default profiles.

[KyleAure]

Thanks, Arjan for the background. I'm new to this spec and didn't know about the prior discussions.
I guess I'm here to offer a fresh perspective as someone who is just now getting involved with this spec. In that regard I agree, what we have today is better than the experience of running the platform TCK.

That being said, I think there is always room for improvement. I also do think there is a value in what you said:

[every test] can be deployed, debugged, and tinkered with standalone, and is isolated from other tests.

However, the downsides to that are we are left with a cluttered project structure with modules containing sparse content, and there is no published artifact that I can pull in and use to test my implementation at the implementation's source. So long as the TCK is written well, implementors should be spending more time resolving issues brought to life by the TCK, then messing around with the TCK itself.

I would argue that since the TCK exists in an open-source project that debugging, and tinkering with the tests is easy enough to accomplish without distributing the source code itself.

[arjantijms]

However, the downsides to that are we are left with a cluttered project structure with modules containing sparse content, and there is no published artifact that I can pull in and use to test my implementation at the implementation's source.

Actually, we do produce a binary artefact, but it's not published to maven central at the moment.

See: https://github.com/jakartaee/faces/blob/master/tck-dist/pom.xml

A locally installed version of that is pulled in by the test runner pom here:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/faces/pom.xml#L63

I would argue that since the TCK exists in an open-source project that debugging, and tinkering with the tests is easy enough to accomplish without distributing the source code itself.

It's not necessarily about having the source or not. It's the fact here that each test is its own war, and it's a standard war that you can deploy everywhere. Some of the other TCKs you mention, like concurrency, create a micro archive within the test, which is then deployed. It's not possible to directly build that war independently using standard tools. It's of course still possible to debug, but grabbing the generated war file from the test for example is far less trivial and sometimes not even useful if Arquillian magic mode (testable mode) is used.

Which brings us to another issue.

The other TCKs are far from a consistent format. Some use the magic mode (like CDI), where the tests are injected with beans which are then tested. The test class is magically (under the hood) transferred to the server to be executed from there. Yet other TCKs build a more or less standard war, deploy that, and communicate with it over HTTP. Still other TCKs (like concurrency) deploy a kind of emulated testng to the server, and a generic test servlet runs tests and only returns true or false.