Open jing-c-tyagi opened 1 year ago
thecampagnards
commented
1 year ago Hello,
What is the RAM size of your machine? You can also use this value https://github.com/camunda-community-hub/camunda-7-community-helm/blob/main/charts/camunda-bpm-platform/values.yaml#L32 to add an extra env and add java OPTS like JAVA_OPTS="-Xms128m -Xmx512m"
jing-c-tyagi
commented
1 year ago JAVA_OPTS="-Xms128m -Xmx512m"
Thanks thecampagnards for your quick response!
we already has resources set in values.yaml file as following, resources: limits: cpu: 1500m memory: 3000Mi requests: cpu: 1200m memory: 2000Mi
based on helm chart: camunda-bpm-platform-7.6.1 with app version of latest, it's java.version = 11.0.15, would set -XX:MaxRAMPercentage be better than JAVA_OPTS="-Xms128m -Xmx512m"? extraEnvs: -XX:MaxRAMPercentage=80.0
thecampagnards
commented
1 year ago Hi, my bad it's catalina can you try something like this CATALINA_OPTS="-Xmx2048m -XX:MaxPermSize=1024m -XX:PermSize=1024m according to this post https://forum.camunda.io/t/java-heap-space-issue/29876/2 ?
helm chart: camunda-bpm-platform-7.6.1 with app version of latest
os: awslinux2 aws eks cluster
What are the steps to reproduce your problem? might related to a runaway task? application failed to function. Could you help to increase the heap size.
What is the problem? $ k logs -n camunda camunda-camunda-bpm-platform-667d56fc9f-lclvl -c camunda-bpm-platform --tail=100
java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3689) ~[na:na] at java.base/java.util.ArrayList.grow(ArrayList.java:238) ~[na:na] at java.base/java.util.ArrayList.grow(ArrayList.java:243) ~[na:na] at java.base/java.util.ArrayList.add(ArrayList.java:486) ~[na:na] at java.base/java.util.ArrayList.add(ArrayList.java:499) ~[na:na] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.restoreProcessInstance(ExecutionEntity.java:1353) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.ensureExecutionTreeInitialized(ExecutionEntity.java:1286) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.ensureProcessInstanceInitialized(ExecutionEntity.java:794) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.getProcessInstance(ExecutionEntity.java:783) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.getRootProcessInstanceId(ExecutionEntity.java:1776) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.initHistoricJobLogEvent(DefaultHistoryEventProducer.java:1167) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.createHistoricJobLogEvt(DefaultHistoryEventProducer.java:1105) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.createHistoricJobLogCreateEvt(DefaultHistoryEventProducer.java:1068) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee]
at org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogManager$1.createHistoryEvent(HistoricJobLogManager.java:187) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.event.HistoryEventProcessor.processHistoryEvents(HistoryEventProcessor.java:77) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogManager.fireJobCreatedEvent(HistoricJobLogManager.java:184) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobManager.insertJob(JobManager.java:81) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.insert(JobEntity.java:165) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobManager.schedule(JobManager.java:114) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.TimerEntity.createNewTimerJob(TimerEntity.java:114) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.TimerEntity.preExecute(TimerEntity.java:97) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:129) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:110) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:43) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:72) ~[camunda-engine-spring-7.17.2-ee.jar!/:7.17.2-ee] at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.20.jar!/:5.3.20] at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:70) ~[camunda-engine-spring-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee]
2023-09-06 18:08:11.429 ERROR 9 — [aTaskExecutor-2] org.camunda.bpm.engine.context : ENGINE-16004 Exception while closing command context: Java heap space
java.lang.OutOfMemoryError: Java heap space at java.base/java.util.HashMap.resize(HashMap.java:700) ~[na:na] at java.base/java.util.HashMap.putVal(HashMap.java:659) ~[na:na] at java.base/java.util.HashMap.put(HashMap.java:608) ~[na:na] at org.camunda.bpm.engine.impl.db.entitymanager.cache.DbEntityCache.putInternal(DbEntityCache.java:204) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.cache.DbEntityCache.putPersistent(DbEntityCache.java:172) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.onEntityLoaded(DbEntityManager.java:261) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.fireEntityLoaded(AbstractPersistenceSession.java:261) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:105) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:182) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:174) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:170) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:154) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionManager.findExecutionsByProcessInstanceId(ExecutionManager.java:113) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.ensureExecutionTreeInitialized(ExecutionEntity.java:1274) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.ensureProcessInstanceInitialized(ExecutionEntity.java:794) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.getProcessInstance(ExecutionEntity.java:783) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.getRootProcessInstanceId(ExecutionEntity.java:1776) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.initHistoricJobLogEvent(DefaultHistoryEventProducer.java:1167) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.createHistoricJobLogEvt(DefaultHistoryEventProducer.java:1105) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.producer.DefaultHistoryEventProducer.createHistoricJobLogCreateEvt(DefaultHistoryEventProducer.java:1068) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee]
at org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogManager$1.createHistoryEvent(HistoricJobLogManager.java:187) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.history.event.HistoryEventProcessor.processHistoryEvents(HistoryEventProcessor.java:77) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogManager.fireJobCreatedEvent(HistoricJobLogManager.java:184) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobManager.insertJob(JobManager.java:81) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.insert(JobEntity.java:165) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobManager.schedule(JobManager.java:114) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.TimerEntity.createNewTimerJob(TimerEntity.java:114) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.TimerEntity.preExecute(TimerEntity.java:97) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:129) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:110) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:43) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee] at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28) ~[camunda-engine-7.17.2-ee.jar!/:7.17.2-ee]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]" 2023-09-06 18:09:28.857 ERROR 9 — [aTaskExecutor-3] org.camunda.bpm.engine.context : ENGINE-16004 Exception while closing command context: Java heap space
java.lang.OutOfMemoryError: Java heap space
2023-09-06 18:09:28.859 ERROR 9 — [nio-8080-exec-5] o.a.coyote.http11.Http11NioProtocol : Failed to complete processing of a request
java.lang.OutOfMemoryError: Java heap space
2023-09-06 18:09:28.860 ERROR 9 — [aTaskExecutor-4] org.camunda.bpm.engine.context : ENGINE-16004 Exception while closing command context: Java heap space
java.lang.OutOfMemoryError: Java heap space
2023-09-06 18:09:28.863 INFO 9 — [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2023-09-06 18:09:28.870 INFO 9 — [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' 2023-09-06 18:09:28.871 INFO 9 — [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 1 ms 2023-09-06 18:09:34.233 WARN 9 — [nio-8080-exec-3] o.a.c.util.SessionIdGeneratorBase : Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [1,800] milliseconds.
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8080-Acceptor"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8080-Poller"