search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-36750] projectActionFloatingBox.jelly: java.lang.OutOfMemoryError: PermGen space #3354

Closed timja closed 7 years ago

timja commented 8 years ago

Jenkins is running out of memory fairly quick and regularly: about once every one or two weeks, probably depending on load.

We've attempted increasing PermGem to no avail and have looked at the processes and threads running while Jenkins was is this state, but could not find any obvious suspects.

At one point we found that generating a test result graph was the culprit and Jelly was mentioned then too. But we're not sure what this error means though.

OS: Linux 3.13.0-88-generic #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux for master and all slave machines.
Java: Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Jenkins: 2.9

hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Ping thread for channel hudson.remoting.Channel@aad0ee5:vmslave14/585) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: PermGen space
        at sun.misc.Unsafe.defineClass(Native Method)
        at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
        at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
        at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
        at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:440)
        at java.lang.Throwable.writeObject(Throwable.java:985)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at hudson.remoting.Command.writeTo(Command.java:87)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:43)
        at hudson.remoting.Channel.send(Channel.java:608)
        at hudson.remoting.Request.callAsync(Request.java:205)

javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.9.jar!/lib/hudson/project/projectActionFloatingBox.jelly:38:74:  java.lang.OutOfMemoryError: PermGen space
    at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:117)
    at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
    at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:104)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.9.jar!/lib/hudson/project/projectActionFloatingBox.jelly:38:74:  java.lang.OutOfMemoryError: PermGen space
    at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:745)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:289)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
    at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
    at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
    ... 69 more
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: PermGen space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2232)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
    at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834)
    at org.kohsuke.stapler.CachingScriptLoader.findScript(CachingScriptLoader.java:62)
    at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:112)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    ... 110 more
Caused by: java.lang.OutOfMemoryError: PermGen space
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at groovy.lang.GroovyClassLoader.access$400(GroovyClassLoader.java:62)
    at groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:500)
    at groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:517)
    at groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:521)
    at org.codehaus.groovy.control.CompilationUnit$16.call(CompilationUnit.java:822)
    at org.codehaus.groovy.control.CompilationUnit$16.call(CompilationUnit.java:830)
    at org.codehaus.groovy.control.CompilationUnit$16.call(CompilationUnit.java:830)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1053)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
    at org.kohsuke.stapler.jelly.groovy.GroovyClassLoaderTearOff.parse(GroovyClassLoaderTearOff.java:92)
    at org.kohsuke.stapler.jelly.groovy.GroovyClassTearOff.parseScript(GroovyClassTearOff.java:50)
    at org.kohsuke.stapler.jelly.groovy.GroovyClassTearOff.parseScript(GroovyClassTearOff.java:39)
    at org.kohsuke.stapler.AbstractTearOff.resolveScript(AbstractTearOff.java:91)
    at org.kohsuke.stapler.jelly.JellyClassTearOff.resolveScript(JellyClassTearOff.java:85)
    at org.kohsuke.stapler.jelly.JellyClassTearOff.resolveScript(JellyClassTearOff.java:47)
    at org.kohsuke.stapler.AbstractTearOff.loadScript(AbstractTearOff.java:97)
    at org.kohsuke.stapler.CachingScriptLoader$1.load(CachingScriptLoader.java:31)
    at org.kohsuke.stapler.CachingScriptLoader$1.load(CachingScriptLoader.java:28)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
Originally reported by ingo, imported from: projectActionFloatingBox.jelly: java.lang.OutOfMemoryError: PermGen space
  • status: Resolved
  • priority: Critical
  • resolution: Won't Fix
  • resolved: 2017-05-23T11:15:05+00:00
  • imported: 2022/01/10
timja commented 8 years ago

danielbeck:

Not enough information to investigate further. Memory dumps, configuration details, etc. would be needed.

JRE 8 may work, there's no separate permgen anymore.

timja commented 7 years ago

danielbeck:

Jenkins now requires Java 8 since April, the next LTS release will require it as well.

Java 8 does not have PermGen space, so this issue is obsolete.