out of memory error messages coming up more frequently

[jdeck88]

I saw this error last week on the NMNH server and now this week on the biscicol server... i've seen the OutofMemoryError message come up when re-deploying a WAR file in jetty in the past. It happened when numerous hot-deployments were done and after time, it seemed that server wanted to be re-started. The fix in this situation was to restart Jetty. This still seems to work. However, this seems to be happening more frequently now and i don't think it necessarily has to do with hot-deployments now... that is, it comes up after the server has been for a time. This makes me think there is a memory leak somewhere.... Here is the message that comes up in the logs.

I wonder if this can be tested by "burning in" a deployment (on a test server of course!) by loading lots of data through some automated process ( i would think the validation components may be the most suspicious scripts)

Caused by: javax.ws.rs.ProcessingException: java.lang.OutOfMemoryError: PermGen space at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at services.RedirectFilter.doFilter(RedirectFilter.java:29) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at biocode.fims.rest.filters.ClickJackFilter.doFilter(ClickJackFilter.java:28) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)

[jdeck88]

And, wondering if this connected in some way to #10

[rodney757]

okay, so looking into this a bit, PermGen stores loaded classes. If classes aren't being unloaded, or they are continually being created, it will eventually throw an OutOfMemoryException. I'm still trying to reproduce this though.

I was able to create a "GC overhead limit exceeded" OutOfMemory error. This was after just 136 iterations of validation and uploading. With over 1800 iterations of just validation, no error occurred leaving me to suspect the uploading code as the cause of the issue.

[rodney757]

I'm not able to reproduce this even after 18000 call to resolver & bcid minter services. Maybe you can reset the access/request log after you redeploy the latest changes. Then next time the OutOfMemoryException occurs, we can look at the logs and have a better idea of what calls might be causing this.

[jdeck88]

Note that i also changed XXMaxPermSize for jetty/java to 1024m ... it wasn't specified earlier and i'm not sure what it was. This might solve the problem. We'll see.

On Sun, Mar 13, 2016 at 6:58 PM, RJ Ewing notifications@github.com wrote:

I'm not able to reproduce this even after 18000 call to resolver & bcid minter services. Maybe you can reset the access/request log after you redeploy the latest changes. Then next time the OutOfMemoryException occurs, we can look at the logs and have a better idea of what calls might be causing this.

— Reply to this email directly or view it on GitHub https://github.com/biocodellc/biocode-fims-commons/issues/11#issuecomment-196104490 .

John Deck (541) 914-4739

[jdeck88]

Based on RJ's evaluation and the change to the XXMaxPermSize on the server, am closing the issue for now.