search

jakartaee / rest

Jakarta RESTful Web Services
Other
355 stars 114 forks source link

TCK: Extend existing TCK or start our own one? #924

Closed mkarg closed 3 years ago

mkarg commented 3 years ago

For JAX-RS 3.1 we need some new TCK tests. We have two choices:

Please all committers, discuss here, so we can come to a conclusion ASAP.

mkarg commented 3 years ago

@arjantijms @chkal @spericas @andymc12 @jansupol @ronsigal @deki Please discuss! :-)

arjantijms commented 3 years ago

For Faces we have essentially agreed to go with the second option. Write new tests in a new TCK.

For Servlet there was discussion about this too, with a hint to try converting the tests. This is an example where I manually converted a handful of tests:

https://github.com/javaee-samples/jakartaee8-tck

All tests are individual projects, use standard JUnit for running the tests and Arquillian for deploying the tests (no further Arquillian magic, Arquillian doesn't take over anything, it only deploys the war to the configured server). That one still uses the original TCK's HTTP client.

For Faces, we more want to move to using an HTTP client like used here: https://github.com/eclipse-ee4j/mojarra/tree/master/test2

The Soteria tests were setup in a similar way as well: https://github.com/eclipse-ee4j/soteria/tree/master/test

Looking at how MicroProfile and Java EE/Jakarta EE samples are set up, they are all fairly similar. There are small differences, like using the unit test or integration test phase, specifying the micro archive that Arquillian uses manually, or taking the full output from the project (latter only possible when doing IT tests), etc.

deki commented 3 years ago

+1 for a TCK renewal, the existing one is just painful to execute

chkal commented 3 years ago

@arjantijms Now I'm curious. Do JSF implementations now have to run and pass two TCKs? The old tests contained in the original Jakarta EE TCK plus the new TCK created for Jakarta Faces?

arjantijms commented 3 years ago

@chkal The plan is to move to just one TCK, the new one.

The matter being that Faces 4.0 (likely version number) is a big change; the deprecated templating language JSP and native managed beans will be removed. This means thousands of tests need to be adjusted.

As you know, JSF (Faces) is by far not as popular anymore as it once was, so effectively we only have two main volunteers who can devote approximately 10% (as a guess) of their time to Faces. In other words we have 0% chance of converting the existing TCK.

mkarg commented 3 years ago

I think it would be a valid option to keep two TCKs for a while. For the Java SE Bootstrap API I could provide tests on a modern base, while we keep the old stuff until we find enough capacity to migrate it. If nobody has a problem with that, I'd like to start coding it next week.

Regarding the HTTP client, I'd really love to use JRE 11's native HTTP client, but I think as long as the TCK must proof that JRE 8 is enough, it is a no-go at least for now.

chkal commented 3 years ago

@chkal The plan is to move to just one TCK, the new one.

Sure, this would be the best case. I just wonder if this goal is realistic. And as you already mentioned, the full migration of all existing tests to "a new TCK" may not be possible for specs like Jakarta Faces, because of the effort that would be required.

Therefore, I agree with @mkarg, that having two TCKs (the old one, which may not be updated very much anymore + the new one, which is used for all new tests). Not sure if there has been any public discussion about this yet.

mkarg commented 3 years ago

@chkal The plan is to move to just one TCK, the new one.

Sure, this would be the best case. I just wonder if this goal is realistic. And as you already mentioned, the full migration of all existing tests to "a new TCK" may not be possible for specs like Jakarta Faces, because of the effort that would be required.

Therefore, I agree with @mkarg, that having two TCKs (the old one, which may not be updated very much anymore + the new one, which is used for all new tests). Not sure if there has been any public discussion about this yet.

As it is up to each spec project to find their own way, this is that public discussion. ;-)

Actually I just started a new TCK for us. Hacking on it right now. :-)

arjantijms commented 3 years ago

And as you already mentioned, the full migration of all existing tests to "a new TCK" may not be possible for specs like Jakarta Faces, because of the effort that would be required.

Yes, so for Faces it's not set in stone yet. If there's a sudden influx of resources (which I doubt, but suppose), we can look at different options.

Also, maybe it's feasible to keep a part of the old TCK, which means we'd end up with two TCKs as well. The idea would be to run the old TCK on Faces 4.0 and whatever breaks we delete. If what remains is deemed to be of sufficient value, we could decide to keep that around.

For the new TCK we'll seed it with around 100 (give or take) tests ported/rewritten from the old TCK, as well as a collection of vendor independent unit and integration tests from both MyFaces and Mojarra.

As for the new Jakarta REST/Jax-rs TCK, I figure it would be best to take a look at the existing MP, CDI, Bean Val and converted JSON-B/P TCKs, as well as to the resources I linked above. Best to have something most inline with those existing solutions.

jansupol commented 3 years ago

Not that anyone cares about the history around the CTS framework nowadays, but it might help a bit to understand the bits how they are now and the complexity behind the CTS framework. Back in the Java EE (5) days, Java EE platform was what mattered most. It was Java EE Spec that defined the containers (the basic are EJB, Servlet, JSP, and appclient - in the CTS terminology those are referred to as "vehicles"). There was a product called CTS that mattered the most, each respective TCK was not as important as the whole platform.

For that reason, the CTS framework supports running the tests in each of the mentioned vehicles, for EJB, it wraps the test in an EJB, deploys it, and there is a client that invokes a JNDI lookup to the test EJB, and the test is executed within the EJB container, and the result is passed to the client. Similarly, it is done for Servlet and other containers. The CTS has to provide a mechanism to allow all this without any need of coding this for each test.

In Jakarta RESTful Web Services tests, we had this only for the API tests. The other tests are bundled just in WARs.

Currently, with all the microservices around, and serverless requirements, the Jakarta EE platform does not seem to be that important as it used to be. The Arquillian based TCK seems fine now, no matter they do not support running the tests in all the vehicles (or do they?).

Jakarta RESTful Web Services TCK consists of about 2800 tests that would be a pain to transform to the Arquillian based tests, though. For that, having only new tests in the Arquillian sounds like the least effort for the test developers. However, that would mean double work for running the tests. I doubt the certification process allows for having two complementary TCK bundles.

Before starting any effort with the new TCK bundle, this should be communicated on the Jakarta EE level, imho.

m0mus commented 3 years ago

I fully support extracting TCKs from CTS into the project repository but I am strongly against having two sets of tests. It must be only one source of truth. You must submit only one set of TCKs for the spec ballot. You can have two sets only temporary while you are developing new functionality and migrating old tests from CTS. I am suggesting to take a look how it was done in JSON Binding and JSON Processing and do it the same way if it makes sense. If not, we should discuss it. The idea is to use the same set of tools for all projects. It will simplify testing multiple implementations. I'll bring this topic to the PMC and Spec Committee this week for guidance.

jansupol commented 3 years ago

I wonder if we could have CTS tests (the old ones) and TCK tests (the new ones). The vendors always used to run both the TCK and the CTS, no matter the tests are the same... That's if the Jakarta EE 10 continues with having the CTS bundle at all...

spericas commented 3 years ago

Regardless of how we structure things internally, we should aim at having a single bundle and a single command to run all the tests; otherwise it would prone to errors and confusion.

arjantijms commented 3 years ago

Thanks @jansupol I'm always interested in the history :)

In practice, maybe the vehicles add some complexity, but IMHO what is a very basic but practical obstacle in the current TCK is that all separate tests are just a list of files that are assembled by an ANT file into for example a war, without any native project structure that an IDE recognises.

In most of the newer setups, each test is a separate application as well, that can be individually deployed, is recognises as a project by all IDEs, and therefore much easier to develop and debug.

I guess people who develop tests for the TCK have/had some personal helper scripts and/or commands to achieve something like this?

jansupol commented 3 years ago

The Eclipse IDE works fine for the TCK repository. It is faster to use just the subset of tests, for JAX-RS TCK instead of all src/com/sun/ts/test tests, the src/com/sun/ts/tests/{common,interop,jaxrs,servlet,signaturetest} folders only. The path is set in the ts.jte file, which is unknown to Eclipse IDE. So it was required to add the jars from the ts.jte classpath to the Eclipse classpath. Not all of them were required, Eclipse works fine with classes that are part of the project but not used by the running code, and cannot be built due to missing dependencies.

The test can be executed from the IDE (if needed, for debugging for instance) when adding the following method to the respective Client.java:

    @Override
    protected com.sun.javatest.Status run(String[] args) {
        if (args.length == 0)
            args = new String[] { "-p", "install/jaxrs/bin/ts.jte", "-t",
                    "<test method name>", "-vehicle", "standalone" };
        return super.run(args);
    }

After the war has been deployed to a running Glassfish, Tomcat, WebLogic, or another server. There were not so many scripts, mostly everything is done by ant clean install, ant deploy, ant run.

FroMage commented 3 years ago

Hi,

As part of our work on RESTEasy Reactive (https://quarkus.io/blog/resteasy-reactive/) we've had to fork the existing Jakarta EE TCK and do the following:

As a result, the entire JAX-RS TCK now runs in less than 3 minutes on RESTEasy Reactive, with each test firing up a new Quarkus process.

It currently sits at https://github.com/quarkusio/quarkus-rest-testsuite

The README is wrong, we didn't update it. Running it is just mvn clean install, but it depends on RESTEasy Reactive as you can see in the https://github.com/quarkusio/quarkus-rest-testsuite/blob/master/tests/pom.xml#L48[pom.xml] file and here's an example test: https://github.com/quarkusio/quarkus-rest-testsuite/blob/master/tests/src/test/java/com/sun/ts/tests/jaxrs/jaxrs21/ee/patch/server/JAXRSClient0179.java#L45

We believe this could be a good basis for a new TCK, provided we can find a way to make this work for other JAX-RS implementations.

scottmarlow commented 3 years ago

At a minimum, the Jakarta EE Platform SPEC requirements still need to be tested. Will the new JAX-RS TCK cover the Platform SPEC requirements?

Or can the new JAX-RS TCK tests be imported to the Platform TCK in some way?

mkarg commented 3 years ago

At a minimum, the Jakarta EE Platform SPEC requirements still need to be tested. Will the new JAX-RS TCK cover the Platform SPEC requirements?

Or can the new JAX-RS TCK tests be imported to the Platform TCK in some way?

Can you please share a link to the exact rules, what a new TCK MUST cover?

jansupol commented 3 years ago

Another effort for the discussion

mkarg commented 3 years ago

Please find a draft for a new TCK in PR #931. Please discuss the general question how to deal with the TCK here but discuss all what really is directly related to that draft there. Thanks.