JsonProviderTest fails if run after other Junits in a test suite

[tabishop]

ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.java will fail if run in a Junit test suite if it executes after any other TCK test that calls JsonProvider.provider(). Junits don't guarantee order of execution of tests within a test suite by default.

This is because the JsonProvider loads the provider as a static final the first time it is requested. The JsonProviderTest.java class attempts to set a system property to a dummy provider class, call JsonProvider.provider(), and then tries to validate that the provider returned is the dummy provider class. However, if any other TCK test that calls JsonProvider.provider() is run before this test, it will fail because the returned provider will be of the class initially used, not the dummy provider.

To recreate, create a Junit Test suite with the JsonProviderTest and any other test that calls JsonProvider.provider() and runs before Json ProviderTest.

This causes problems when trying to develop test automation for Open Liberty support of JSONP 2.1

[KyleAure]

Here is the stack trace of this failure:

org.opentest4j.AssertionFailedError: expected: <ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.DummyJsonProvider> but was: <org.eclipse.parsson.JsonProviderImpl>
    at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
    at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
    at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124)
    at ee.jakarta.tck.jsonp.api.provider.JsonProviderTest.systemProperty(JsonProviderTest.java:57)

[lukasj]

TCK user guide currently does not allow having both test suites on the classpath at the same time or better - this setup is not documented/supported

[KyleAure]

@lukasj I am confused by this statement. The TCK user guide makes no mention of the JsonProviderTest needed to be run separately from any of the other tests in the jakarta.json-tck-tests-2.1.0.jar test archive. Can you please point me to the section that specifies this? Note, that this test exists in the TCK test archive, not TCK pluggability which also has it's own provider.

[lukasj]

Right, my bad, I have misread it as you mean running both groups at once (which is with or in the UG)

current runner runs each test (=testclass) in its own VM (as does the platform tck I believe), so the problem is not visible there. Do you think this has to made explicit somewhere? There is no way to avoid test interference without it.

[KyleAure]

Hey Lukas, I think if this is a requirement then there should be a mention of it in the TCK User Guide. Especially since Jakarta as a whole has no consistency in requirements for Standalone TCK Requirements.