tomaswolf / gerrit-gitblit-plugin

Integrates standard GitBlit (currently 1.7.1) as a repository viewer in Gerrit
Apache License 2.0
22 stars 5 forks source link

Running into memory problems #24

Closed edschnir closed 8 years ago

edschnir commented 8 years ago

I am using Gerrit 2.12,2 and GitBlit plugin 2.171.1.2 and after a while of running getting the following error:

[2016-08-08 17:46:44,561] [HTTP-249] WARN  com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot build BloomFilter for jdbc:h2:file:///local/gerrit/cache/conflicts: Deserialization failed, cause: "java.lang.ClassNotFoundException: com.google.gerrit.extensions.common.SubmitType" [90027-176]
[2016-08-08 13:31:23,903] [HTTP-1761] ERROR org.apache.wicket.Session : Exception when detaching/serializing page java.lang.OutOfMemoryError: PermGen space
[2016-08-08 13:42:26,872] [HTTP-854] ERROR org.apache.wicket.Session : Exception when detaching/serializing page java.lang.StackOverflowError

And here is a more complete error:

[2016-08-09 08:00:52,132] [HTTP-481] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:00:54,373] [HTTP-538] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:03:30,018] [HTTP-539] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:03:32,146] [HTTP-538] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:03:46,126] [HTTP-567] INFO  com.google.gerrit.httpd.auth.ldap.LdapLoginServlet : 'salodha' failed to sign in: Incorrect username or password
[2016-08-09 08:04:17,415] [HTTP-565] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:04:25,520] [HTTP-566] ERROR org.apache.wicket.RequestCycle : Can't instantiate page using constructor public com.gitblit.wicket.pages.TreePage(org.apache.wicket.PageParame
ters) and argument r = "[mfp/nso-scratch.git]" h = "[package-library-installer]"
org.apache.wicket.WicketRuntimeException: Can't instantiate page using constructor public com.gitblit.wicket.pages.TreePage(org.apache.wicket.PageParameters) and argument r = "[mfp/nso-s
cratch.git]" h = "[package-library-installer]"
        at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:212)
        at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:89)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.newPage(BookmarkablePageRequestTarget.java:305)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.getPage(BookmarkablePageRequestTarget.java:320)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.processEvents(BookmarkablePageRequestTarget.java:234)
        at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
        at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1279)
        at org.apache.wicket.RequestCycle.step(RequestCycle.java:1358)
        at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1465)
        at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
        at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
        at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
        at com.googlesource.gerrit.plugins.gitblit.GerritWicketFilter.doFilter(GerritWicketFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:236)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:138)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor218.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:188)
        ... 61 more
Caused by: java.lang.OutOfMemoryError: PermGen space
[2016-08-09 08:04:30,831] [HTTP-539] ERROR org.apache.wicket.RequestCycle : Can't instantiate page using constructor public com.gitblit.wicket.pages.LogPage(org.apache.wicket.PageParamet
ers) and argument r = "[mfp/nso-scratch.git]" h = "[package-library-installer]"
org.apache.wicket.WicketRuntimeException: Can't instantiate page using constructor public com.gitblit.wicket.pages.LogPage(org.apache.wicket.PageParameters) and argument r = "[mfp/nso-sc
ratch.git]" h = "[package-library-installer]"
        at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:212)
        at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:89)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.newPage(BookmarkablePageRequestTarget.java:305)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.getPage(BookmarkablePageRequestTarget.java:320)
        at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.processEvents(BookmarkablePageRequestTarget.java:234)
        at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
        at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1279)
        at org.apache.wicket.RequestCycle.step(RequestCycle.java:1358)
        at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1465)
        at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
        at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
        at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
        at com.googlesource.gerrit.plugins.gitblit.GerritWicketFilter.doFilter(GerritWicketFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:236)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:138)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:188)
        ... 61 more
Caused by: java.lang.OutOfMemoryError: PermGen space

With the following configuration (gitblit.properties at $GERRIT_HOME/etc):

#
# GitBlit custom settings
#

# Disable GitBlit Git Server, as Gerrit already serves Git repositories
git.cacheRepositoryList = false
git.enableGitServlet = false
git.onlyAccessBareRepositories = true

# Points to Gerrit Git repositories folder
git.repositoriesFolder = /git/sjc-apl-grt10

# Disable cache repository list
git.cacheRepositoryList = false

# Disable GitBlit's SSH daemon. Gerrit has one already.
git.sshPort = 0

# The ticket service would need the git servlet. Anyway, this is functionality
# provided by Gerrit.
tickets.service = com.googlesource.gerrit.plugins.gitblit.app.ReallyNullTicketService

# Disable the fanout service
fanout.port = 0

# Allow the user of GitBlit cookie authentication for avoiding overload on
# Gerrit authentication every time that a repository is browsed
web.allowCookieAuthentication = true

# Requires users to authenticate against Gerrit before giving any access
web.authenticateViewPages = true

# Disable GitBlit Web administration, as it supposed to be made on Gerrit init
web.allowAdministration = false
web.authenticateAdminPages = true

# Disable the ability to fork a repository
web.allowForking = false

# Disable GitBlit RPC services
web.enableRpcServlet = false
web.enableRpcManagement = false
web.enableRpcAdministration = false

# No Commit Cache (default 14)
web.activityCacheDays = 0

# Disable the 'mount' of GitBlit parameters in the URL, it would mess-up the
# Gerrit plugins URLs generation and parsing of them when Gerrit repos have '/'
# in their name. Additionally avoid the usage of URL-encoded '/' (%2F) as it may
# be automatically decoded by reverse proxies and then even forbidden for security
# reasons.
web.mountParameters = false

web.allowZipDownloads = true
web.compressedDownloads = zip gz

Also added the following arguments to Gerrit Java in order to allow GC and unload of PermGen:

 -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
tomaswolf commented 8 years ago

Give your JVM a bit more PermGen space, or move to newer JVM. This is described at https://github.com/tomaswolf/gerrit-gitblit-plugin#installation .

edschnir commented 8 years ago

Thank you for your prompt answer. We are using JDK 1.7.0_67 As per your instructions I have added the following to teh Gerrit container configuration:

-XX:PermSize=512m -XX:MaxPermSize=1g -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

And here are all the data of that Java process:

-bash-4.1$ ps -ef | grep Gerrit
gerrit2   2500 23868  0 01:06 pts/1    00:00:00 grep Gerrit
gerrit2  29640     1 28 00:57 pts/0    00:02:21 GerritCodeReview -Dcom.sun.management.jmxremote.port=49999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:PermSize=512m -XX:MaxPermSize=1g -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Xmx42g -jar /local/gerrit/bin/gerrit.war daemon -d /local/gerrit --run-id=1471247866.29590
-bash-4.1$ /local/java/jdk1.7.0_67/bin/jinfo -flag PermSize 29640
-XX:PermSize=536870912
-bash-4.1$ /local/java/jdk1.7.0_67/bin/jinfo 29640
Attaching to process ID 29640, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
Java System Properties:

com.sun.management.jmxremote.authenticate = false
java.runtime.name = Java(TM) SE Runtime Environment
java.vm.version = 24.65-b04
sun.boot.library.path = /local/java/jdk1.7.0_67/jre/lib/amd64
java.vendor.url = http://java.oracle.com/
java.vm.vendor = Oracle Corporation
path.separator = :
java.rmi.server.randomIDs = true
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
sun.os.patch.level = unknown
sun.java.launcher = SUN_STANDARD
user.country = US
user.dir = /local/gerrit
java.vm.specification.name = Java Virtual Machine Specification
com.sun.management.jmxremote.port = 49999
org.apache.sshd.common.io.IoServiceFactoryFactory = org.apache.sshd.common.io.mina.MinaServiceFactoryFactory
java.runtime.version = 1.7.0_67-b01
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
os.arch = amd64
java.endorsed.dirs = /local/java/jdk1.7.0_67/jre/lib/endorsed
line.separator =

java.io.tmpdir = /tmp
java.vm.specification.vendor = Oracle Corporation
os.name = Linux
sun.jnu.encoding = UTF-8
java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.specification.name = Java Platform API Specification
java.class.version = 51.0
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
os.version = 2.6.32-504.12.2.el6.x86_64
user.home = /home/gerrit2
user.timezone = America/Los_Angeles
java.awt.printerjob = sun.print.PSPrinterJob
file.encoding = UTF-8
java.specification.version = 1.7
user.name = gerrit2
java.class.path = /local/gerrit/bin/gerrit.war
java.vm.specification.version = 1.7
sun.arch.data.model = 64
sun.java.command = /local/gerrit/bin/gerrit.war daemon -d /local/gerrit --run-id=1471247866.29590
java.home = /local/java/jdk1.7.0_67/jre
user.language = en
java.specification.vendor = Oracle Corporation
awt.toolkit = sun.awt.X11.XToolkit
com.sun.management.jmxremote.ssl = false
java.vm.info = mixed mode
java.version = 1.7.0_67
java.ext.dirs = /local/java/jdk1.7.0_67/jre/lib/ext:/usr/java/packages/lib/ext
sun.boot.class.path = /local/java/jdk1.7.0_67/jre/lib/resources.jar:/local/java/jdk1.7.0_67/jre/lib/rt.jar:/local/java/jdk1.7.0_67/jre/lib/sunrsasign.jar:/local/java/jdk1.7.0_67/jre/lib/jsse.jar:/local/java/jdk1.7.0_67/jre/lib/jce.jar:/local/java/jdk1.7.0_67/jre/lib/charsets.jar:/local/java/jdk1.7.0_67/jre/lib/jfr.jar:/local/java/jdk1.7.0_67/jre/classes
java.awt.headless = true
java.vendor = Oracle Corporation
file.separator = /
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
sun.io.unicode.encoding = UnicodeLittle
sun.cpu.endian = little
sun.cpu.isalist =

VM Flags:

-Dcom.sun.management.jmxremote.port=49999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:PermSize=512m -XX:MaxPermSize=1g -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Xmx42g
tomaswolf commented 8 years ago

Did that solve the problem?

edschnir commented 8 years ago

Seems like yes. It have been running without any memory problem since then. Thank you.