search

cioos-siooc / cioos-pacific-erddap

CIOOS Pacific ERDDAP main repository used to maintain, develop and integrate ERDDAP datasets.
https://data.cioospacific.ca/erddap
2 stars 0 forks source link

erddap crashes a lot #5

Closed n-a-t-e closed 1 year ago

n-a-t-e commented 1 year ago

Lots of out of memory errors in the logs, here are some examples:

Using default messages.xml from  /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml
CfToFromGcmd static loading /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt
*** EDStatic Low Level Startup finished successfully.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Heap dump file created [4494748240 bytes in 16.452 secs]
Exception in thread "http-nio-8080-Poller" Exception in thread "http-nio-8080-Acceptor" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
11-Mar-2023 11:18:27.363 SEVERE [http-nio-8080-exec-6] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
    java.lang.OutOfMemoryError: Java heap space
11-Mar-2023 11:18:27.365 SEVERE [Catalina-utility-1] org.apache.catalina.core.ContainerBase.threadStart Exception processing background thread
    java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.apache.catalina.core.ContainerBase.threadStart(ContainerBase.java:1288)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor.run(ContainerBase.java:1334)
--
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:833)
    Caused by: java.lang.OutOfMemoryError: Java heap space
11-Mar-2023 11:18:27.365 SEVERE [Catalina-utility-2] org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async timeouts
    java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.apache.coyote.AbstractProtocol.startAsyncTimeout(AbstractProtocol.java:603)
        at org.apache.coyote.AbstractProtocol.lambda$start$0(AbstractProtocol.java:588)
--
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:833)
    Caused by: java.lang.OutOfMemoryError: Java heap space
11-Mar-2023 11:18:31.157 INFO [http-nio-8080-exec-6] org.apache.catalina.connector.CoyoteAdapter.checkRecycled Encountered a non-recycled request and recycled it forcedly.
    org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
        at org.apache.catalina.connector.CoyoteAdapter.checkRecycled(CoyoteAdapter.java:540)
        at org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1429)
--
Using default messages.xml from  /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml
CfToFromGcmd static loading /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt
*** EDStatic Low Level Startup finished successfully.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Unable to create java_pid1.hprof: File exists
Exception in thread "http-nio-8080-Poller" Exception in thread "http-nio-8080-Acceptor" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
14-Mar-2023 13:33:41.069 SEVERE [Catalina-utility-1] org.apache.catalina.core.ContainerBase.threadStart Exception processing background thread
    java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.apache.catalina.core.ContainerBase.threadStart(ContainerBase.java:1288)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor.run(ContainerBase.java:1334)
--
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:833)
    Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:732)
        at java.base/java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:151)
        at java.base/sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
        at java.base/java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:370)
--
Using default messages.xml from  /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml
CfToFromGcmd static loading /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt
*** EDStatic Low Level Startup finished successfully.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Unable to create java_pid1.hprof: File exists
groupadd: group 'tomcat' already exists
ERDDAP Running with:  -server -Xms6G -Xmx6G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true  -DerddapContentDirectory=/usr/local/tomcat/content/erddap/ -Djna.tmpdir=/tmp/ -Djava.security.egd=file:/dev/./urandom
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
n-a-t-e commented 1 year ago

Server upgrade resolved this so far