Bug Report: Spring Applications Deploy Incorrectly When Using Docker Compose / FISH-6368

[njohnson111]

Spring Boot maven web apps cannot run in payara/server-full:5.2022.2-jdk11 docker container

---

When I run the app in local payara, the url http://localhost/springapp/test works fine, but when I deploy my app on dockerized payara, I get HTTP Status 404 - Not Found. In dockerized payara administration panel I can see that the app has been deployed and when I run asadmin list-applications in cli I can see that the app has been deployed as well.

Expected Outcome

Spring app without 404 errors.

Current Outcome

Error 404 on every request of every Spring app.

Steps to reproduce

Just deploy a simple hello-world spring application with web module and thymeleaf template engine.

Hint that will speed up the debugging process

When I run the application on docker, dispatcherServlet and messageDispatcherServlet are not loaded, but when I run the app locally, the servlets are loaded.

Environment

• Distribution: payara/server-full:5.2022.2-jdk11 docker image
• JDK Version: 11
• Operating System: the os used from your docker image

[njohnson111]

Logs from the container:

[Entrypoint] running /opt/payara/scripts/init_1_generate_deploy_commands.sh
Adding deployment target /opt/payara/deployments/springapp.war to post boot commands

[Entrypoint] ignoring /opt/payara/scripts/init.d/*

Executing Payara Server with the following command line:
/usr/lib/jvm/zulu11-ca-amd64/bin/java
-cp
/opt/payara/appserver/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.logging/java.util.logging=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-XX:NewRatio=2
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=70.0
-Xbootclasspath/a:/opt/payara/appserver/glassfish/lib/grizzly-npn-api.jar
-Xss512k
-server
-javaagent:/opt/payara/appserver/glassfish/lib/monitor/flashlight-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=/opt/payara/appserver/glassfish/domains/domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=/opt/payara/appserver/glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=/opt/payara/appserver/glassfish
-Dfelix.fileinstall.poll=5000
-Djava.security.policy=/opt/payara/appserver/glassfish/domains/domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=/opt/payara/appserver/glassfish/domains/domain1/config/keystore.jks
-Djava.security.auth.login.config=/opt/payara/appserver/glassfish/domains/domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=/opt/payara/appserver/glassfish/domains/domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-Djdk.attach.allowAttachSelf=true
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djdbc.drivers=org.h2.Driver
-Dorg.glassfish.grizzly.nio.DefaultSelectorHandler.force-selector-spin-detection=true
-Djava.library.path=/opt/payara/appserver/glassfish/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-prebootcommandfile
/opt/payara/config/pre-boot-commands-final.asadmin
-upgrade
false
-read-stdin
true
-postbootcommandfile
/opt/payara/config/post-boot-commands-final.asadmin
-domainname
domain1
-domaindir
/opt/payara/appserver/glassfish/domains/domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--user,,,admin,,,--passwordfile,,,/opt/payara/passwordFile,,,--secure=false,,,--terse=false,,,--extraterse=false,,,--echo=false,,,--interactive=false,,,--autoname=false,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,/opt/payara/appserver/glassfish/domains,,,domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
/opt/payara/appserver/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false

Launching Payara Server on Felix platform
May 30, 2022 8:36:06 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner createBundleProvisioner
INFO: Create bundle provisioner class = class com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.
Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishRuntime@7a378326 in service registry.
Reading in commandments from /opt/payara/config/pre-boot-commands-final.asadmin
#!## LogManagerService.postConstruct : rootFolder=/opt/payara/appserver/glassfish
#!## LogManagerService.postConstruct : templateDir=/opt/payara/appserver/glassfish/lib/templates
#!## LogManagerService.postConstruct : src=/opt/payara/appserver/glassfish/lib/templates/logging.properties
#!## LogManagerService.postConstruct : dest=/opt/payara/appserver/glassfish/domains/domain1/config/logging.properties
[#|2022-05-30T20:36:11.341+0000|INFO|Payara 5.2022.2|javax.enterprise.logging|_ThreadID=28;_ThreadName=RunLevelControllerThread-1653942971189;_TimeMillis=1653942971341;_LevelValue=800;_MessageID=NCLS-LOGGING-00009;|
  Running Payara Version: Payara Server  5.2022.2 #badassfish (build 306)|#]

[#|2022-05-30T20:36:11.350+0000|INFO|Payara 5.2022.2|javax.enterprise.logging|_ThreadID=28;_ThreadName=RunLevelControllerThread-1653942971189;_TimeMillis=1653942971350;_LevelValue=800;_MessageID=NCLS-LOGGING-00010;|
  Server log file is using Formatter class: com.sun.enterprise.server.logging.ODLLogFormatter|#]

[#|2022-05-30T20:36:11.816+0000|INFO|Payara 5.2022.2|org.glassfish.ha.store.spi.BackingStoreFactoryRegistry|_ThreadID=30;_ThreadName=RunLevelControllerThread-1653942971199;_TimeMillis=1653942971816;_LevelValue=800;|
  Registered fish.payara.ha.hazelcast.store.HazelcastBackingStoreFactoryProxy for persistence-type = hazelcast in BackingStoreFactoryRegistry|#]

[#|2022-05-30T20:36:14.412+0000|INFO|Payara 5.2022.2|fish.payara.nucleus.hazelcast.HazelcastCore|_ThreadID=28;_ThreadName=RunLevelControllerThread-1653942971189;_TimeMillis=1653942974412;_LevelValue=800;|
  Hazelcast Instance Bound to JNDI at payara/Hazelcast|#]

[#|2022-05-30T20:36:14.413+0000|INFO|Payara 5.2022.2|fish.payara.nucleus.hazelcast.HazelcastCore|_ThreadID=28;_ThreadName=RunLevelControllerThread-1653942971189;_TimeMillis=1653942974413;_LevelValue=800;|
  JSR107 Caching Provider Bound to JNDI at payara/CachingProvider|#]

Logs from Payara:

[2022-05-30T20:36:06.293+0000] [] [INFO] [NCLS-GFLAUNCHER-00005] [javax.enterprise.launcher] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1653942966293] [levelValue: 800] [[
  JVM invocation command line:
/usr/lib/jvm/zulu11-ca-amd64/bin/java
-cp
/opt/payara/appserver/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.logging/java.util.logging=ALL-UNNAMED
--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-XX:NewRatio=2
-XX:+UseContainerSupport
-XX:MaxRAMPercentage=70.0
-Xbootclasspath/a:/opt/payara/appserver/glassfish/lib/grizzly-npn-api.jar
-Xss512k
-server
-javaagent:/opt/payara/appserver/glassfish/lib/monitor/flashlight-agent.jar
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=/opt/payara/appserver/glassfish/domains/domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=/opt/payara/appserver/glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=/opt/payara/appserver/glassfish
-Dfelix.fileinstall.poll=5000
-Djava.security.policy=/opt/payara/appserver/glassfish/domains/domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=/opt/payara/appserver/glassfish/domains/domain1/config/keystore.jks
-Djava.security.auth.login.config=/opt/payara/appserver/glassfish/domains/domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=/opt/payara/appserver/glassfish/domains/domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dorg.jboss.weld.serialization.beanIdentifierIndexOptimization=false
-Dosgi.shell.telnet.ip=127.0.0.1
-Djdk.attach.allowAttachSelf=true
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djdbc.drivers=org.h2.Driver
-Dorg.glassfish.grizzly.nio.DefaultSelectorHandler.force-selector-spin-detection=true
-Djava.library.path=/opt/payara/appserver/glassfish/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-prebootcommandfile
/opt/payara/config/pre-boot-commands-final.asadmin
-upgrade
false
-read-stdin
true
-postbootcommandfile
/opt/payara/config/post-boot-commands-final.asadmin
-domainname
domain1
-domaindir
/opt/payara/appserver/glassfish/domains/domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--user,,,admin,,,--passwordfile,,,/opt/payara/passwordFile,,,--secure=false,,,--terse=false,,,--extraterse=false,,,--echo=false,,,--interactive=false,,,--autoname=false,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,/opt/payara/appserver/glassfish/domains,,,domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
/opt/payara/appserver/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false]]

Proof of successful deployment:

[JamesHillyard]

Hi @njohnson111,

Please can you provide a simple to follow scenario how to reproduce this including the simple hello-world spring application you mention and instructions on how to reproduce this issue? A reproducer should ideally follow the SSCCE rules: http://www.sscce.org/. It will greatly help us to find the cause and fix it.

Thanks, James

[njohnson111]

Sure, here are the steps:

STEP 1

I used Amazon Corretto version 11.0.10 because it is earlier version of your dockerized payara's version, so it should work.

STEP 2

STEP 3

STEP 4

IndexController.java

package com.example.helloworld;

import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.RequestMapping;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class IndexController {
    @RequestMapping("/")
    public String indexPage(ModelMap model, HttpServletRequest request, HttpServletResponse response) {
        return "index";
    }
}

STEP 5

STEP 6

index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Hello World</title>
</head>
<body>
Hello World!
</body>
</html>

STEP 7

The hello-world app is working locally:

STEP 8

Now run a payara/server-full:5.2022.2-jdk11 container and deploy the *.war file of step 7 (automatically by putting it into $DEPLOY_DIR or manually via the admin panel)

STEP 9

Open the url http://localhost:8080/helloworld/ and you will get 404 error.

HINT

In dockerized payara there are missing components:

CONCLUSION

My claim is that no spring app can work on this container so please free to test and your own spring apps. There is no need to test mine.

Thank you!

[njohnson111]

Hello again!

Was my last post clear? Do you want additional information?

I am just asking because the status remained: "Waiting on the issue requester to give more details or share a reproducer"

Thank you.

[svendiedrichsen]

@njohnson111 As I interpret this roadmap ticket Payara currently does not support Spring applications: https://github.com/payara/Payara/issues/4564

[njohnson111]

@njohnson111 As I interpret this roadmap ticket Payara currently does not support Spring applications: #4564

Thank you very much for your answer @svendiedrichsen. I do not understand exactly what @smillidge means in his post, but in localhosted Payara, spring applications run normally. If Payara doesn't support Spring at all, should not there be a problem everywhere? Now the problem is only with this particular docker image. Also (personal opinion) what is the reason for having a java app server that does not support the java's best framework?

@JamesHillyard can you please tell me if you managed to reproduce the issue and if there is a case to fix it so that I do not wait in vain; Thank you again.

[AlanRoth]

Hi @njohnson111,

@JamesHillyard will tend to your issue given time. We never provide a guaranteed time frame for investigation for community issues or fixes, if you require a expeditated and prioritized investigation then a Payara Enterprise license is always available.

Thanks for your understanding, Alan

[njohnson111]

Hello @AlanRoth,

Thanks for your reply. I did not ask for an "investigation" but to find out why the status "pending" is left for days while I gave all the details. I also asked, without of course requiring it with timeline, to deploy a spring application to this particular docker image to see if it has a problem or not, since there are many of us who have a problem. I think that a deployment process takes a little less than 3 seconds plus 30 seconds for the creation of a hello-world spring application.

However, I still understand that you may not even have 33 seconds for the community version. So, is there an enterprise payara jdk11 docker image that can run spring applications so I can buy a license? Or should I buy it first and then have the investigation done by you?

Thank you again.

[JamesHillyard]

Hi @njohnson111,

My apologies for the confusion, I have been Out-Of-Office and therefore not had time to see your most recent comments including reproducer instructions and get a reply back to you.

I have been able to successfully deploy a Spring Application based on the instructions you provided to a payara/server-full:5.2022.2-jdk11 docker container. On my docker environment, I am seeing all the expected components as seen on a standard payara install and I am able to successfully load the page at http://localhost:8080/helloworld-0.0.1-SNAPSHOT

For reference, I am using Windows 10, Maven 3.6.3 and Azul JDK 11.0.11.

Unfortunately I'm not able to reproduce the issue you have raised, if you can provide an alternative reproducer I would be happy to attempt to reproduce this issue again.

Best Regards, James

[njohnson111]

Hello @JamesHillyard,

Thanks for your reply.

No problem.

Well, the differences I see are that you used Azul JDK instead of Corretto JDK, and Maven version 3.6.3 instead of version 3.8.1 which is used.

Since I couldn't find the version you used on azul website, I used version 11.0.15+10 that I found here:

https://www.azul.com/downloads/?version=java-11-lts&os=windows&architecture=x86-64-bit&package=jdk

And I created a new project with Spring's Initializr to test with azul:

tomcat (working fine):

payara/server-full:5.2022.2-jdk11 (not working):

So, despite the fact that dispatcherServlet was loaded this time, it is still not working.

This thing drives me insane. I would say it's my fault, but since it appears in others, something else is probably to blame.

Could you kindly upload your war file here so that I may deploy it and test it on my setup?

[JamesHillyard]

Hi @njohnson111,

The JDK and Maven version discrepancy shouldn't cause a problem, but please do try my reproducer on your machine and see if it works for you.

Unfortunately GitHub doesn't allow the upload of .war files directly and access to the entire reproducer may prove useful to you so I have uploaded it to this repo: https://github.com/JamesHillyard/5783_Reproducer. The war can be found in the target directory.

After deployment, navigating to http://localhost:8080/helloworld-0.0.1-SNAPSHOT should result in the page successfully loading.

Thanks, James

[njohnson111]

Hello @JamesHillyard .

Thank you for your fast response.

The JDK and Maven version discrepancy shouldn't cause a problem, but please do try my reproducer on your machine and see if it works for you.

I could not agree more!!!

Unfortunately GitHub doesn't allow the upload of .war files directly and access to the entire reproducer may prove useful to you so I have uploaded it to this repo: https://github.com/JamesHillyard/5783_Reproducer. The war can be found in the target directory.

Yes, it was very useful. I confirm that your .war worked and I am so sorry for not being clear at step 5.

index.html should be in the templates folder, otherwise you "bypass" thymeleaf

So, here is the correct structure:

I created and a fork with the change:

https://github.com/njohnson111/5783_Reproducer

[JamesHillyard]

Hi @njohnson111,

Thank you for confirming that and making changes to the reproducer, it's much appreciated.

I was able to successfully reach the endpoint 6/8 times on a regular Payara Server configuration, twice getting a 404 and seeing the incorrect modules and components page, and 8/8 times using the docker image. Each time I restarted the server (or docker container), recompiled and deployed the updated reproducer based on your fork.

Is the 404 error consistently reproducible for you?

Thanks, James

[njohnson111]

Hello @JamesHillyard ,

Thank you for your fast response again. I m glad that you were able to reproduce the issue and you got the 404 error.

On a regular Payara Server I had no problem on any of my tests. On dockerized Payara Server I am getting 404 all the time. Even if I refresh the page 40 times I will get this:

Whitelabel Error Page

This application has no explicit mapping for /error, so you are seeing this as a fallback.
Tue Jun 14 21:28:58 EEST 2022
There was an unexpected error (type=Not Found, status=404).

The only thing that does change is the current time.

and 8/8 times using the docker image

Excuse me, do you mean 8/8 times you got 404 or 8/8 times you got successful response?

[JamesHillyard]

Hi @njohnson111,

Thank you for confirming this, my apologies if I was unclear, 8/8 times I got a successful response using the docker image. If you create a new docker image, deploy the spring application and test it, do you always get a 404 response?

Thanks, James

[njohnson111]

Hello @JamesHillyard

I will try again from the scratch:

STEP1

docker run -p 8080:8080 -p 4848:4848 payara/server-full:5.2022.2-jdk11

STEP2

a new payara/server-full:5.2022.2-jdk11 container got created:

STEP3

latest reproducer https://github.com/njohnson111/5783_Reproducer.git cloned here: C:\Projects\latestReproducer

STEP4

new project from existing sources

STEP5

loaded

STEP6

STEP7

build artifacts

STEP8

C:\Projects\latestReproducer\target\helloworld-0.0.1-SNAPSHOT.war got created

STEP9

https://localhost:4848/common/index.jsf

STEP10

checked components

STEP11

http://localhost:8080/helloworld-0.0.1-SNAPSHOT and... it worked this time :/ ... (I don't know why lol. I suspect it may be to blame that I was creating the project with intellij while you are not but I will move on with this reproducer for now.)

---

Ok, so lets try with docker-compose which is our main objective:

STEP1:

version: "3.8"
services:
  springApp:
    image: payara/server-full:5.2022.2-jdk11
    container_name: springApp
    volumes:
      - "./target/helloworld-0.0.1-SNAPSHOT.war:/opt/payara/deployments/helloworld-0.0.1-SNAPSHOT.war"
    ports:
      - "4848:4848"
      - "8080:8080"
      - "8181:8181"

STEP2: failed to deploy

[#|2022-06-15T13:06:48.987+0000|INFO|Payara 5.2022.2|javax.enterprise.web|_ThreadID=1;_ThreadName=main;_TimeMillis=1655298408987;_LevelValue=800;| 2022-06-15T13:06:48.987915900Z WebModule[/helloworld-0.0.1-SNAPSHOT] ServletContext.log():2 Spring WebApplicationInitializers detected on classpath|#] 2022-06-15T13:06:48.987919800Z 2022-06-15T13:06:49.146287200Z [#|2022-06-15T13:06:49.145+0000|SEVERE|Payara 5.2022.2|javax.enterprise.web.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1655298409145;_LevelValue=1000;| 2022-06-15T13:06:49.146310400Z Error invoking ServletContainerInitializer org.springframework.web.SpringServletContainerInitializer 2022-06-15T13:06:49.146313200Z java.lang.NoClassDefFoundError: fish/payara/logging/PayaraLogManager 2022-06-15T13:06:49.146315200Z at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.(GlassFishMain.java:79) 2022-06-15T13:06:49.146317300Z at java.base/java.lang.Class.forName0(Native Method) 2022-06-15T13:06:49.146319300Z at java.base/java.lang.Class.forName(Class.java:315) 2022-06-15T13:06:49.146321100Z at org.springframework.boot.SpringApplication.deduceMainApplicationClass(SpringApplication.java:277) 2022-06-15T13:06:49.146323000Z at org.springframework.boot.SpringApplication.(SpringApplication.java:269) 2022-06-15T13:06:49.146325400Z at org.springframework.boot.builder.SpringApplicationBuilder.createSpringApplication(SpringApplicationBuilder.java:129) 2022-06-15T13:06:49.146327200Z at org.springframework.boot.builder.SpringApplicationBuilder.(SpringApplicationBuilder.java:102) 2022-06-15T13:06:49.146329100Z at org.springframework.boot.builder.SpringApplicationBuilder.(SpringApplicationBuilder.java:98) 2022-06-15T13:06:49.146331100Z at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createSpringApplicationBuilder(SpringBootServletInitializer.java:166) 2022-06-15T13:06:49.146333000Z at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:130) 2022-06-15T13:06:49.146334900Z at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:97) 2022-06-15T13:06:49.146337200Z at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:174) 2022-06-15T13:06:49.146340100Z at org.apache.catalina.core.StandardContext.callServletContainerInitializers(StandardContext.java:5852) 2022-06-15T13:06:49.146354500Z at com.sun.enterprise.web.WebModule.callServletContainerInitializers(WebModule.java:736) 2022-06-15T13:06:49.146359800Z at org.apache.catalina.core.StandardContext.start(StandardContext.java:5750) 2022-06-15T13:06:49.146363900Z at com.sun.enterprise.web.WebModule.start(WebModule.java:619) 2022-06-15T13:06:49.146367800Z at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:958) 2022-06-15T13:06:49.146371300Z at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:941) 2022-06-15T13:06:49.146374900Z at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694) 2022-06-15T13:06:49.146378500Z at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1871) 2022-06-15T13:06:49.146382000Z at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1623) 2022-06-15T13:06:49.146385400Z at com.sun.enterprise.web.WebApplication.start(WebApplication.java:107) 2022-06-15T13:06:49.146388600Z at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123) 2022-06-15T13:06:49.146392300Z at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:292) 2022-06-15T13:06:49.146396100Z at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:361) 2022-06-15T13:06:49.146399800Z at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:623) 2022-06-15T13:06:49.146406600Z at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:597) 2022-06-15T13:06:49.146409000Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) 2022-06-15T13:06:49.146410900Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) 2022-06-15T13:06:49.146412600Z at java.base/java.security.AccessController.doPrivileged(Native Method) 2022-06-15T13:06:49.146414400Z at java.base/javax.security.auth.Subject.doAs(Subject.java:361) 2022-06-15T13:06:49.146416200Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) 2022-06-15T13:06:49.146418000Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) 2022-06-15T13:06:49.146419800Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) 2022-06-15T13:06:49.146421600Z at java.base/java.security.AccessController.doPrivileged(Native Method) 2022-06-15T13:06:49.146423400Z at java.base/javax.security.auth.Subject.doAs(Subject.java:361) 2022-06-15T13:06:49.146425100Z at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) 2022-06-15T13:06:49.146426900Z at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) 2022-06-15T13:06:49.146428700Z at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120) 2022-06-15T13:06:49.146434200Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) 2022-06-15T13:06:49.146436100Z at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755) 2022-06-15T13:06:49.146437900Z at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.executeCommand(CommandExecutorImpl.java:183) 2022-06-15T13:06:49.146440400Z at com.sun.enterprise.admin.cli.embeddable.CommandExecutorImpl.run(CommandExecutorImpl.java:96) 2022-06-15T13:06:49.146442300Z at fish.payara.boot.runtime.BootCommand.execute(BootCommand.java:69) 2022-06-15T13:06:49.146444100Z at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:141) 2022-06-15T13:06:49.146445800Z at fish.payara.boot.runtime.BootCommands.executeCommands(BootCommands.java:135) 2022-06-15T13:06:49.146447600Z at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.doBootCommands(GlassFishMain.java:306) 2022-06-15T13:06:49.146449400Z at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:145) 2022-06-15T13:06:49.146451200Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2022-06-15T13:06:49.146453000Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2022-06-15T13:06:49.146454800Z at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2022-06-15T13:06:49.146456600Z at java.base/java.lang.reflect.Method.invoke(Method.java:566) 2022-06-15T13:06:49.146458500Z at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:121) 2022-06-15T13:06:49.146461700Z at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54) 2022-06-15T13:06:49.146463600Z Caused by: java.lang.ClassNotFoundException: fish.payara.logging.PayaraLogManager not found by fish.payara.server.internal.core.glassfish [349] 2022-06-15T13:06:49.146465400Z at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1585) 2022-06-15T13:06:49.146467300Z at org.apache.felix.framework.BundleWiringImpl.access$300(BundleWiringImpl.java:79) 2022-06-15T13:06:49.146469000Z at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1970) 2022-06-15T13:06:49.146470800Z at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) 2022-06-15T13:06:49.146472600Z ... 54 more 2022-06-15T13:06:49.146474400Z |#] 2022-06-15T13:06:49.146476200Z 2022-06-15T13:06:49.146611300Z [#|2022-06-15T13:06:49.146+0000|SEVERE|Payara 5.2022.2|javax.enterprise.web.core|_ThreadID=1;_ThreadName=main;_TimeMillis=1655298409146;_LevelValue=1000;_MessageID=AS-WEB-CORE-00174;| 2022-06-15T13:06:49.146622000Z Startup of context /helloworld-0.0.1-SNAPSHOT failed due to previous errors|#]

I updated my repo https://github.com/njohnson111/5783_Reproducer.git with docker-compose.yml

Thanks a lot and sorry for the endless "documentation"!

[JamesHillyard]

Hi @njohnson111,

Thanks for your detailed reply, I have followed it through again and here are my findings:

and... it worked this time :/ ... (I don't know why lol. I suspect it may be to blame that I was creating the project with intellij while you are not but I will move on with this reproducer for now.)

At this stage it also worked for me - It's worth noting I also built the project with IntelliJ Idea Ultimate 2022.2.1 each time I built a reproducer.

This makes me wonder if the error is inconsistently occurring or specifically requires docker compose to occur and deploying spring applications does work when deploying via the admin console (As I did on my 8 previous successful runs).

---

I then went on to use the docker compose file you provided and I was finally able to reproduce the issue! http://localhost:8080/helloworld-0.0.1-SNAPSHOT took me to a 404 page and the deployment information was incorrectly displayed.

Would you be able to confirm this error doesn't occur when using the admin console to deploy the application when using the docker image consistently on your end please? It would greatly help us scope this issue.

Thanks, James

[njohnson111]

Hello @JamesHillyard,

Thanks for your fast response once again.

It's worth noting I also built the project with IntelliJ Idea Ultimate 2022.2.1 each time I built a reproducer.

Oh... We have the same version then...

I then went on to use the docker compose file you provided and I was finally able to reproduce the issue! http://localhost:8080/helloworld-0.0.1-SNAPSHOT took me to a 404 page and the deployment information was incorrectly displayed.

Great news!!! We found the problem after all!

Would you be able to confirm this error doesn't occur when using the admin console to deploy the application when using the docker image consistently on your end please? It would greatly help us scope this issue.

Yes of course! I will perform several tests tomorrow and I will let you know the result.

I'd also like to ask you a question that isn't totally relevant. Do you know if a docker image for version 17 will be released soon? I ask because maybe I could use it until the problem with version 11 is solved.

Thank you again for your time and support.

[JamesHillyard]

Hi @njohnson111,

Thank you, that's much appreciated, I will await the results of your testing.

Do you know if a docker image for version 17 will be released soon?

Yes! A JDK 17 based docker image will be available in the next release of Payara Community!

Thanks, James

[njohnson111]

Hello @JamesHillyard,

Thank you for responding and informing me about version 17.

I created a new project only to make sure your reproducer was not different from the one I had when I reported the problem.

The new project was successfully deployed and tested more than seven times, and I also experimented with other scenarios (for example I disabled and enabled the project, I un-deployed and redeployed it etc).

There was absolutely no issue from admin panel. As a result, the issue appears solely with compose.

So, I guess my "job" is over at this point. : )

Thank you once again.

PS: Although I know what it's like to work on a huge project like this project, and I highly respect your work, I can't help but wonder if you were able to focus somewhere with the information you have so far, and if you have an unofficial non-binding estimate of how long the solution may take? : D

[JamesHillyard]

Hi @njohnson111,

Thank you for sharing those details so quickly, It's much appreciated! In this case I will do some testing to confirm if there is a wider issue here with using compose & the Payara Docker Image, or if this is only applicable to spring applications. For now at least we have found a workaround in using the admin console, however I appreciate this isn't ideal.

Our efforts are currently focused on Jakarta EE 10 and Payara 6, so at this time I can't provide an estimate on how long a fix will take, however we always encourage you to contribute, so if you wish you can submit a PR with a fix and we will gladly review it.

After confirming if this is a spring exclusive issue or there is a larger bug here with using docker compose and the Payara Docker image, I will update you and accept this issue.

Many thanks for your patience and cooperation, James

[njohnson111]

Hello @JamesHillyard

Thanks for your answer.

I totally understand...

Well, I'm afraid that my knowledge is not enough to make corrections in such a project, so I will wait for an update! : )

Regards!

[JamesHillyard]

Hi @njohnson111,

I tested the same scenario but using a standard Jakarta EE application using docker compose and everything worked as expected so this seems to be a spring only issue as you reported. I have raised an internal issue FISH-6368 to address this bug, we will pick up the issue at some point in the future.

Edit: I have updated the title to more accurately reflects the issue we have identified

Thank you very much for your cooperation in finding a reproducer for this issue, it is much appreciated.

Thanks, James

[njohnson111]

Hello @JamesHillyard !

I'm glad I "helped" and I agree with the change of title.

I'll keep an eye on this thread for the solution.

Warm regards till then.