search

SORMAS-Foundation / SORMAS-Project

SORMAS (Surveillance, Outbreak Response Management and Analysis System) is an early warning and management system to fight the spread of infectious diseases.
https://sormas.org
GNU General Public License v3.0
292 stars 140 forks source link

Report `EarClassLoader` redeployment problem with newer Payara [2] #7145

Open StefanKock opened 2 years ago

StefanKock commented 2 years ago

Problem Description

Coming from #2511 and https://github.com/hzi-braunschweig/SORMAS-Project/issues/4601#issuecomment-916167447 .

We can only deploy the SORMAS artifacts once on a Payara domain newer than payara-5.194. After the second deployment of sormas-ear, the EJB remote lookup/instantiation fails because the classes to use come from different EarClassLoader instances.

Proposed Solution

  1. Provide a commit/branch where one can easily replay the problem
  2. Report to Payara team what our problem is
  3. Maybe provide a stripped down version of SORMAS to illustrate the problem with less code.

Possible Alternatives

Accept that redeployment does not work.

Additional Information

StefanKock commented 2 years ago

Current idea: Use https://github.com/hzi-braunschweig/SORMAS-Docker#quick-start and provide a reproducable way to show the problem. (on branch development, because master is still on v1.66.4)

Unfortunately it does not start on my machine:

docker-compose up
error while interpolating services.base-keycloak.mem_limit: failed to cast to expected type: strconv.ParseInt: parsing "1024M": invalid syntax

docker-compose up
error while interpolating services.base-postgres.mem_limit: failed to cast to expected type: strconv.ParseInt: parsing "2000M": invalid syntax

docker-compose up
error while interpolating services.base-keycloak-postgres.mem_reservation: failed to cast to expected type: strconv.ParseInt: parsing "1000M": invalid syntax

docker-compose up
error while interpolating services.base-apache2.mem_reservation: failed to cast to expected type: strconv.ParseInt: parsing "200M": invalid syntax

docker compose version
Docker Compose version v2.2.3

This seems to be a known and fixed problem, but is not yet released: https://github.com/docker/compose/issues/9095

Workaround: Append these values in the .env file:

APPSERVER_JVM_MAX=4096000000
APPSERVER_MEM=4300000000
APPSERVER_MEM_RESERVED=1500000000
WEBSERVER_MEM=1000000000
WEBSERVER_MEM_RESERVED=200000000

KEYCLOAK_MEM=1024000000
KEYCLOAK_MEM_RESERVED=512000000
DB_MEM=2000000000
DB_MEM_RESERVED=1000000000

KEYCLOAK_DB_MEM=2000000000
KEYCLOAK_DB_MEM_RESERVED=1000000000

DB_DUMP_MEM=500000000
DB_DUMP_MEM_RESERVED=100000000
StefanKock commented 2 years ago

Reproducing problem with SORMAS-Docker:

# Checkout repo
git clone https://github.com/hzi-braunschweig/SORMAS-Docker.git
cd SORMAS-Docker
### We want to test 1.67.0+ with payara-5.2021.10
git checkout development

# 1. Start SORMAS (first sync of images will take some minutes)
docker-compose up

# 2. Watch log in a separate console 
docker logs -f sormas-docker-sormas-1

# 3. Go into container with running SORMAS-Project
docker exec -ti sormas-docker-sormas-1 bash
cd /opt/domains/sormas/autodeploy

# 4. Redeploy of a sormas WAR works (try several times if you want)
rm sormas-rest.war
cp ../deployments/sormas-rest.war .

# 5. Redeploy sormas-ear.ear works in itself
rm sormas-ear.ear
cp ../deployments/sormas-ear.ear .

# 6. Produce exception: Repeating step 4 results in exception and autodeploy failing
rm sormas-rest.war
cp ../deployments/sormas-rest.war .

# Note: It seems that at some point health checks kick in and restart sormas-docker-sormas-1 indefinitely
Resulting exception (StackTrace) ``` sormas-docker-sormas-1 | [2022-02-02T09:44:19.425+0100] [Payara 5.2021.10] [SEVERE] [AS-WEB-CORE-00108] [javax.enterprise.web.core] [tid: _ThreadID=109 _ThreadName=payara-executor-service-scheduled-task] [timeMillis: 1643791459425] [levelValue: 1000] [[ sormas-docker-sormas-1 | ContainerBase.addChild: start: sormas-docker-sormas-1 | org.apache.catalina.LifecycleException: javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB sormas-docker-sormas-1 | at org.apache.catalina.core.StandardContext.start(StandardContext.java:5778) sormas-docker-sormas-1 | at com.sun.enterprise.web.WebModule.start(WebModule.java:619) sormas-docker-sormas-1 | at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:958) sormas-docker-sormas-1 | at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:941) sormas-docker-sormas-1 | at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694) sormas-docker-sormas-1 | at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1851) sormas-docker-sormas-1 | at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1603) sormas-docker-sormas-1 | at com.sun.enterprise.web.WebApplication.start(WebApplication.java:107) sormas-docker-sormas-1 | at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123) sormas-docker-sormas-1 | at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:292) sormas-docker-sormas-1 | at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:361) sormas-docker-sormas-1 | at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:623) sormas-docker-sormas-1 | at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:592) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552) sormas-docker-sormas-1 | at java.base/java.security.AccessController.doPrivileged(Native Method) sormas-docker-sormas-1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:361) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574) sormas-docker-sormas-1 | at java.base/java.security.AccessController.doPrivileged(Native Method) sormas-docker-sormas-1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:361) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120) sormas-docker-sormas-1 | at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoOperation.run(AutoOperation.java:149) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoDeployer.deploy(AutoDeployer.java:532) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoDeployer.deployAll(AutoDeployer.java:437) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:371) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:362) sormas-docker-sormas-1 | at org.glassfish.deployment.autodeploy.AutoDeployService$1.run(AutoDeployService.java:231) sormas-docker-sormas-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) sormas-docker-sormas-1 | at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) sormas-docker-sormas-1 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) sormas-docker-sormas-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) sormas-docker-sormas-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) sormas-docker-sormas-1 | at java.base/java.lang.Thread.run(Thread.java:829) sormas-docker-sormas-1 | Caused by: javax.ejb.EJBException: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:399) sormas-docker-sormas-1 | at com.sun.ejb.containers.BaseContainer.getContext(BaseContainer.java:2607) sormas-docker-sormas-1 | at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:2024) sormas-docker-sormas-1 | at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:204) sormas-docker-sormas-1 | at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79) sormas-docker-sormas-1 | at com.sun.proxy.$Proxy964.getAuthenticationProvider(Unknown Source) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) sormas-docker-sormas-1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:143) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:173) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatchToServant(ServerRequestDispatcherImpl.java:528) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.protocol.ServerRequestDispatcherImpl.dispatch(ServerRequestDispatcherImpl.java:199) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.protocol.MessageMediatorImpl.handleRequestRequest(MessageMediatorImpl.java:1565) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.protocol.SharedCDRClientRequestDispatcherImpl.marshalingComplete(SharedCDRClientRequestDispatcherImpl.java:128) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.protocol.ClientDelegateImpl.invoke(ClientDelegateImpl.java:258) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:188) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:140) sormas-docker-sormas-1 | at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:226) sormas-docker-sormas-1 | at de.symeda.sormas.api.__ConfigFacade_Remote_DynamicStub.getAuthenticationProvider(de/symeda/sormas/api/__ConfigFacade_Remote_DynamicStub.java) sormas-docker-sormas-1 | at de.symeda.sormas.api._ConfigFacade_Wrapper.getAuthenticationProvider(de/symeda/sormas/api/_ConfigFacade_Wrapper.java) sormas-docker-sormas-1 | at de.symeda.sormas.rest.RestConfig$FilterStartupListener.contextInitialized(RestConfig.java:101) sormas-docker-sormas-1 | at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:5178) sormas-docker-sormas-1 | at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:681) sormas-docker-sormas-1 | at org.apache.catalina.core.StandardContext.start(StandardContext.java:5756) sormas-docker-sormas-1 | ... 37 more sormas-docker-sormas-1 | Caused by: javax.ejb.EJBException: javax.ejb.CreateException: Could not create stateless EJB sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:653) sormas-docker-sormas-1 | at com.sun.ejb.containers.util.pool.NonBlockingPool.getObject(NonBlockingPool.java:219) sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer._getContext(StatelessSessionContainer.java:395) sormas-docker-sormas-1 | ... 62 more sormas-docker-sormas-1 | Caused by: javax.ejb.CreateException: Could not create stateless EJB sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:473) sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer.access$000(StatelessSessionContainer.java:100) sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer$SessionContextFactory.create(StatelessSessionContainer.java:651) sormas-docker-sormas-1 | ... 64 more sormas-docker-sormas-1 | Caused by: java.lang.IllegalStateException: Exception attempting to inject Env-Prop: de.symeda.sormas.backend.common.ConfigFacadeEjb/props@Field-Injectable Resource. Class name = de.symeda.sormas.backend.common.ConfigFacadeEjb Field name=props@java.lang.String@@@ into class de.symeda.sormas.backend.common.ConfigFacadeEjb: Can not set java.util.Properties field de.symeda.sormas.backend.common.ConfigFacadeEjb.props to de.symeda.sormas.backend.common.ConfigFacadeEjb sormas-docker-sormas-1 | at org.glassfish.weld.services.InjectionServicesImpl.aroundInject(InjectionServicesImpl.java:194) sormas-docker-sormas-1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:46) sormas-docker-sormas-1 | at org.jboss.weld.injection.producer.DefaultInjector.inject(DefaultInjector.java:73) sormas-docker-sormas-1 | at org.jboss.weld.module.ejb.DynamicInjectionPointInjector.inject(DynamicInjectionPointInjector.java:61) sormas-docker-sormas-1 | at org.jboss.weld.module.ejb.SessionBeanInjectionTarget.inject(SessionBeanInjectionTarget.java:138) sormas-docker-sormas-1 | at org.glassfish.weld.services.JCDIServiceImpl$JCDIInjectionContextImpl.inject(JCDIServiceImpl.java:581) sormas-docker-sormas-1 | at org.glassfish.weld.services.JCDIServiceImpl.injectEJBInstance(JCDIServiceImpl.java:328) sormas-docker-sormas-1 | at com.sun.ejb.containers.BaseContainer.injectEjbInstance(BaseContainer.java:1818) sormas-docker-sormas-1 | at com.sun.ejb.containers.StatelessSessionContainer.createStatelessEJB(StatelessSessionContainer.java:437) sormas-docker-sormas-1 | ... 66 more sormas-docker-sormas-1 | Caused by: com.sun.enterprise.container.common.spi.util.InjectionException: Exception attempting to inject Env-Prop: de.symeda.sormas.backend.common.ConfigFacadeEjb/props@Field-Injectable Resource. Class name = de.symeda.sormas.backend.common.ConfigFacadeEjb Field name=props@java.lang.String@@@ into class de.symeda.sormas.backend.common.ConfigFacadeEjb: Can not set java.util.Properties field de.symeda.sormas.backend.common.ConfigFacadeEjb.props to de.symeda.sormas.backend.common.ConfigFacadeEjb sormas-docker-sormas-1 | at com.sun.enterprise.container.common.impl.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:638) sormas-docker-sormas-1 | at com.sun.enterprise.container.common.impl.util.InjectionManagerImpl.inject(InjectionManagerImpl.java:439) sormas-docker-sormas-1 | at com.sun.enterprise.container.common.impl.util.InjectionManagerImpl.injectInstance(InjectionManagerImpl.java:153) sormas-docker-sormas-1 | at org.glassfish.weld.services.InjectionServicesImpl.aroundInject(InjectionServicesImpl.java:184) sormas-docker-sormas-1 | ... 74 more sormas-docker-sormas-1 | Caused by: java.lang.IllegalArgumentException: Can not set java.util.Properties field de.symeda.sormas.backend.common.ConfigFacadeEjb.props to de.symeda.sormas.backend.common.ConfigFacadeEjb sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58) sormas-docker-sormas-1 | at java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:75) sormas-docker-sormas-1 | at java.base/java.lang.reflect.Field.set(Field.java:780) sormas-docker-sormas-1 | at com.sun.enterprise.container.common.impl.util.InjectionManagerImpl._inject(InjectionManagerImpl.java:594) sormas-docker-sormas-1 | ... 77 more ```
StefanKock commented 2 years ago

Reported as bug: https://github.com/payara/Payara/issues/5597

StefanKock commented 2 years ago

Reply from payara team: My reproducer was rejected because it's too complex and uses not an official payara docker image. In addition, the auto-deployment feature is discouraged for production use and therefore probably not used in official docker images.

StefanKock commented 1 year ago

This ticket should be rethinked since Payara 6 is out and Payara 5 is said to be end of community support at the end of 2022. https://www.payara.fish/downloads/payara-platform-community-edition/

StefanKock commented 1 year ago

Tested with payara-5.2022.5 (#10499), redeployment of sormas-ear still fails with the following exception:

org.hibernate.boot.InvalidMappingException: Could not parse mapping document: C:\srv\payara-domains\sormas_de\eclipseApps\sormas-ear\sormas-backend_jar\META-INF\listener.xml (INPUT_STREAM)
    at org.hibernate.boot.jaxb.internal.InputStreamXmlSource.doBind(InputStreamXmlSource.java:46)
...
Caused by: org.apache.xerces.impl.dv.DVFactoryException: DTD factory class org.apache.xerces.impl.dv.dtd.DTDDVFactoryImpl does not extend from DTDDVFactory.
    at org.apache.xerces.impl.dv.DTDDVFactory.getInstance(Unknown Source)
MartinWahnschaffe commented 1 year ago

Also failes with exploded archives. PS: Had to set <unpack>true</unpack> for the sormas-backend module in the sormas-ear maven ear plugin.