Bug Report: The Arquiliian test with Payara Embedded All failed/FISH-7730

[charleech]

Brief Summary

Cannot test the simple CDI bean with JUnit5, Arquillian, arquillian-payara-server-embedded and payara-embedded-all.

The system shows the error as the following: -

java.lang.reflect.InaccessibleObjectException: 
    Unable to make field private static final java.util.HashMap sun.net.www.protocol.jar.JarFileFactory.fileCache 
    accessible: module java.base does not "opens sun.net.www.protocol.jar" to unnamed module

Then I had added the JVM options as --add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED, there were failures to add more and more, then I ended up with the following:-

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED"

The Arquillian archive had been deployed, but there was another failure as the following: -

org.glassfish.deployment.common.DeploymentException: 
    CDI deployment failure:WELD-001524: Unable to load proxy class for bean 
    Built-in Bean [jakarta.transaction.UserTransaction] with qualifiers [@Default] 
    with class interface jakarta.transaction.UserTransaction 
    -- WELD-001524: Unable to load proxy class for bean Built-in 
        Bean [jakarta.transaction.UserTransaction] with qualifiers [@Default] 
        with class interface jakarta.transaction.UserTransaction

Expected Outcome

The Arquilian archive have to be deployed to the payara-embedded-all and the test have to be success.

Current Outcome

By using the following: -

• fish.payara.arquillian:arquillian-payara-server-embedded version 3.0.alpha8
• fish.payara.extras:payara-embedded-all version 6.2023.7

There is a failure as the following: -

Jul 28, 2023 7:37:08 PM org.glassfish.kernel.event.EventsImpl send
WARNING: Exception while dispatching an event
java.lang.reflect.InaccessibleObjectException: Unable to make field private static final java.util.HashMap sun.net.www.protocol.jar.JarFileFactory.fileCache accessible: module java.base does not "opens sun.net.www.protocol.jar" to unnamed module @768b970c
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
        at com.sun.enterprise.admin.util.JarFileUtils.closeCachedJarFiles(JarFileUtils.java:77)
        at fish.payara.nucleus.util.CleanupPostBoot.event(CleanupPostBoot.java:78)
        at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
        at com.sun.enterprise.v3.server.AppServerStartup.postStartupJob(AppServerStartup.java:434)
        at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:357)
        at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:237)
        at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:219)
        at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:84)
        at fish.payara.arquillian.container.payara.embedded.PayaraContainer.start(PayaraContainer.java:202)
        at org.jboss.arquillian.container.impl.ContainerImpl.start(ContainerImpl.java:179)
        at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:137)
        at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:133)
        at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:208)
        at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startContainer(ContainerLifecycleController.java:133)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)

Then I had added the JVM options as --add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED, there were failures to add more and more, then I ended up with the following:-

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED"

The Arquillian archive had been deployed, but there was another failure as the following: -

SEVERE: Exception during lifecycle processing
org.glassfish.deployment.common.DeploymentException: CDI deployment failure:WELD-001524: Unable to load proxy class for bean Built-in Bean [jakarta.transaction.UserTransaction] with qualifiers [@Default] with class interface jakarta.transaction.UserTransaction -- WELD-001524: Unable to load proxy class for bean Built-in Bean [jakarta.transaction.UserTransaction] with qualifiers [@Default] with class interface jakarta.transaction.UserTransaction
        at org.jboss.weld.bean.proxy.ProxyFactory.getProxyClass(ProxyFactory.java:409)
        at org.jboss.weld.bean.proxy.ProxyFactory.run(ProxyFactory.java:360)
        at org.jboss.weld.bean.proxy.ProxyFactory.create(ProxyFactory.java:352)
        at org.jboss.weld.bean.builtin.ee.AbstractEEBean.<init>(AbstractEEBean.java:36)
        at org.jboss.weld.module.jta.UserTransactionBean.<init>(UserTransactionBean.java:52)
        at org.jboss.weld.module.jta.WeldTransactionsModule.preBeanRegistration(WeldTransactionsModule.java:46)
        at org.jboss.weld.module.WeldModules.preBeanRegistration(WeldModules.java:158)
        at org.jboss.weld.bootstrap.BeanDeployment.createBeans(BeanDeployment.java:224)
        at org.jboss.weld.bootstrap.WeldStartup.deployBeans(WeldStartup.java:435)
        at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:87)
        at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:581)
        at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:437)
        at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
        at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:577)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:613)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:376)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:376)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at com.sun.enterprise.admin.cli.embeddable.DeployerImpl.deploy(DeployerImpl.java:131)
        at com.sun.enterprise.admin.cli.embeddable.DeployerImpl.deploy(DeployerImpl.java:104)
        at fish.payara.arquillian.container.payara.embedded.PayaraContainer.deploy(PayaraContainer.java:244)

Reproducer

I've created the reproducer at the following: -

• https://github.com/charleech/payara6-evaluation/tree/main/simple-cdi

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED"

mvn clean install -Darq-mode=payara-embedded

Note

With Payara Micro: Succcess

• fish.payara.arquillian:arquillian-payara-micro-managed version 3.0.alpha8
• fish.payara.extraspayara-microversion6.2023.7`

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
--add-opens=java.base/java.io=ALL-UNNAMED"

mvn clean install -Darq-mode=payara-micro

With Payara Managed: Success

• fish.payara.arquillian:arquillian-payara-server-managed version 3.0.alpha8
• fish.payara.distributions:payara-ml:zip version 6.2023.7

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
--add-opens=java.base/java.io=ALL-UNNAMED"

mvn clean install

Operating System

Windows 11 Home Single Language 22H2 22621.1992

JDK Version

openjdk version "17.0.8" 2023-07-18 OpenJDK Runtime Environment Temurin-17.0.8+7 (build 17.0.8+7) OpenJDK 64-Bit Server VM Temurin-17.0.8+7 (build 17.0.8+7, mixed mode, sharing)

Payara Distribution

Payara Embedded All

[charleech]

Surprisingly, it works fine with JDK 11 !

JDK

openjdk version "11.0.20" 2023-07-18 OpenJDK Runtime Environment Temurin-11.0.20+8 (build 11.0.20+8) OpenJDK 64-Bit Server VM Temurin-11.0.20+8 (build 11.0.20+8, mixed mode)

parent pom.xml

<my.java.version>11</my.java.version>
<maven.compiler.release>${my.java.version}</maven.compiler.release>
<maven.compiler.source>${my.java.version}</maven.compiler.source>
<maven.compiler.target>${my.java.version}</maven.compiler.target>

First trial

There is a failure as the following: -

Jul 29, 2023 10:53:33 AM com.sun.appserv.ClassLoaderUtil releaseLoader
WARNING: classloaderutil.errorReleasingLoader
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.ArrayList jdk.internal.loader.URLClassPath.loaders accessible: module java.base does not "opens jdk.internal.loader" to unnamed module @89959b
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
        at com.sun.appserv.ClassLoaderUtil.getField(ClassLoaderUtil.java:301)
        at com.sun.appserv.ClassLoaderUtil.initForClosingJars(ClassLoaderUtil.java:283)
        at com.sun.appserv.ClassLoaderUtil.init(ClassLoaderUtil.java:261)
        at com.sun.appserv.ClassLoaderUtil.releaseLoader(ClassLoaderUtil.java:137)
        at com.sun.appserv.ClassLoaderUtil.releaseLoader(ClassLoaderUtil.java:109)
        at org.glassfish.web.loader.WebappClassLoader.stop(WebappClassLoader.java:2032)
        at org.glassfish.web.loader.WebappClassLoader.preDestroy(WebappClassLoader.java:1997)
        at org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:293)
        at org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:235)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:568)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:613)

Second trial

export JAVA_TOOL_OPTIONS=--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED

The build is success !

[charleech]

After searching through the internet with trial and error, I'm able to make it works with JDK 17 by adding the JVM argument as the following: -

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
-Djdk.attach.allowAttachSelf=true \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED"

mvn clean install -Darq-mode=payara-embedded -am

[charleech]

Even the test is success, the stopping process is too slow. It takes around 2+ minutes.

I also have a chance to dig deeper and found that the slowness is from the method named undeploy of the fish.payara.arquillian.container.payara.embedded.PayaraContainer (branch = Payara6).

https://github.com/payara/ecosystem-arquillian-connectors/blob/Payara6/payara-embedded/src/main/java/fish/payara/arquillian/container/payara/embedded/PayaraContainer.java#L283

    @Override
    public void undeploy(Archive<?> archive) throws DeploymentException {
        try {
            glassfish.getDeployer().undeploy(createDeploymentName(archive.getName())); // <---- This line !
        } catch (Exception e) {
            throw new DeploymentException("Could not undeploy " + archive.getName(), e);
        }
    }

Again, the slowness only occurs for JDK 17. While the JDK 11 has no any issue and quite fast as usual.

[charleech]

I have a chance to add the failure test case as the following: -

    @Test
    @Order(2)
    @DisplayName(
        "When greeting as failure by Jakarta EE should be failed."
    )
    void whenGreetAsFailure() {
        String expected = null;
        String actual   = null;

        expected = "Hello JUnit";

        actual   = this.greeter.greet("UNKNOWN");

        BDDAssertions.then(actual).
            as("The greeting must be valid.").
            isNotNull().
            isNotEmpty().
            isEqualTo(expected);
    }

The expected is, there should 1 failure case as the same from payara-micro

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   GreeterIT.whenGreetAsFailure:83 [The greeting must be valid.]
expected: "Hello JUnit"
 but was: "Hello UNKNOWN"
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

but the result from payara-embedded-all is all success !

[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]

Furthermore, I also have tried to print debug for both test methods, but there is no any printed on the console. It looks like they are not executed.

[felixif]

Hello @charleech,

I have tested the reproducer according to your instructions, and I can see the issue with the test being shown as passed rather than failed. Just to understand the problem: you are saying that the same tests in Payara Micro give the correct results (one pass one fail), compared to Payara Embedded which passes both of the tests (which I can confirm)?

[charleech]

Yes, correct.

Regarding to the README

To test against the payara-micro please use the following command: -

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
-Djdk.attach.allowAttachSelf=true \
--add-opens=java.base/java.io=ALL-UNNAMED"

mvn clean install -Darq-mode=payara-micro

To test against the payara-embedded-all please use the following command: -

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
-Djdk.attach.allowAttachSelf=true \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED \
--add-opens=java.base/java.net=ALL-UNNAMED \
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED"

mvn clean install -Darq-mode=payara-embedded

[charleech]

Furthermore, testing against the payara-ml is also an expected as the same as payara-micro

To test against the payara-ml (aka managed) please use the following command: -

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true \
-Djdk.attach.allowAttachSelf=true \
--add-opens=java.base/java.io=ALL-UNNAMED"

mvn clean install

Note:

• There is no any -Dxxx since it is a default active Maven profile.
• The in-container logs file will be located at
  • simple-cdi/target/container/payara6/glassfish/domains/domain1/logs/server.log

[github-actions[bot]]

Greetings, It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please? We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[charleech]

Hi @felixif

Do you need any information to re-produce ?

[felixif]

Hello @charleech,

No, apologies for the automation that set the issue as abandoned. The reproducer you provided and the instructions are excellent. I was able to confirm that payara-micro provides the correct test results, and payara-embedded passes both tests, but in my case, the same is true for payara-ml (which in your case behaves like payara-micro).

Nevertheless, I have created an internal issue FISH-7730, and our development team is going to have a look as soon as they can. I mentioned the discrepancy between our test results in the case of payara-ml, in case they will have to take a deeper look at both payara-ml andpayara-embedded.

Thank you very much for reporting this bug!

Best regards, Felix