Closed tsuna closed 7 years ago
Not good.
The above only tells me there's some problem with Apache Wicket (which is used by Gitblit) trying to serialize some page. With only that, I have no chance of tracking this down.
Can you give me more clues? Is there anything more in the log? Any hint which page it might have been? Anything in the log just before or after? Any clues from Gerrit's httpd log about the actions that occurred at about the same time?
And -- does it also occur with the latest version?
Just upgraded to Gerrit version 2.12.2 and installed gitblit plugin version 2.12.171.1 and see same Exception:
[2016-07-15 20:19:39,068] [HTTP-582] ERROR org.apache.wicket.Session : Exception when detaching/serializing page
java.lang.StackOverflowError
at java.lang.Exception.
So it occurs also with the latest version. But that stack trace still gives no hint at what the source of the problem might be.
See my questions above. And one additional question: what the symptoms on the UI?
Same over here but just with the really big repositories, larger than 1 Gb and 60,000 commits. No symptoms other than the nasty error message in the log, nothing in the UI
This is truly annoying. Any hint which page it might be?
Took a long look at the Gitblit internals for the umpteenth time. I don't see anything obvious, though I do see some suspicious things:
What causes trouble for Java serialization are long chains of object references. (Circular references should be no problem.) RevCommit in particular has links to its parent commits. GitBlit frequently constructs lists of RevCommits (for instance, history, or activity). Chances are good that in such a list all entries are not just in the list, but additionally linked through the parent links. Serializing such a list will recursively walk the whole parent chain in the JGit RevCommit objects upon serializing the first commit. If the parent chains are long enough, this might well lead to a StackOverflowError.
I'm not sure where the actual bug is now: JGit or Gitblit? Should the parent links in a JGit RevCommit even be serialized, or should they be transient? (There is one more link in a JGit RevCommit that definitely should not be serialized; it's an internal hash collision chain link.)
In any case, Gitblit should take care not to try to serialize JGit objects when its pages get serialized. Fixing that is not going to be easy.
Note that all this is so far a conjecture only, but it could fit the stack traces shown here: RevCommit has the parent links as "RevCommit[] parents" (zero, one, or two), which would match the java.io.ObjectOutputStream.writeArray invocations shown in the traces.
For now, try switching off that activity panel. Set web.activityCacheDays
in gitblit.properties
to zero. That should at least de-activate one place where this might occur.
Indeed the following snippet reproduces a stack trace very similar to the ones shown here:
/**
* Read up to nofCommits commits from a repository; then try to serialize the very first
* commit read.
*/
public static void failingSerialize(File dotGitDir, int nofCommits) throws IOException {
Repository repo = FileRepositoryBuilder.create(dotGitDir);
try (RevWalk walk = new RevWalk(repo)) {
walk.markStart(walk.parseCommit(repo.resolve(Constants.HEAD)));
RevCommit commit = walk.next();
if (commit == null) {
System.out.println("No commit found."); //$NON-NLS-1$
} else {
RevCommit[] history = new RevCommit[nofCommits];
history[0] = commit;
for (int i = 1; i < nofCommits; i++) {
commit = walk.next();
if (commit == null) {
System.out.println("Nothing: " + i); //$NON-NLS-1$
break;
} else {
history[i] = commit;
}
}
try (ByteArrayOutputStream bs = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bs)) {
oos.writeObject(history[0]);
System.out.println("Serialized"); //$NON-NLS-1$
}
}
}
}
Serializing the first commit will in fact attempt to recursively serialize all commits read. Ka-boom. (When exactly it crashes depends on the stack size; for me it crashes on the gitblit repo just short of 2000 commits.)
Gitblit does similar things in a few places, and even though it does in general pay attention not to load thousands of commits, it may well be that it ends up with similar parent commit chains. I've put together a draft release of the plugin that works around this issue and that tries to ensure that Wicket pages never try to serialize JGit RevCommits.
Please give v2.12.171.4 a try, and please do report back if that helped. It's a bit a shot in the dark as I have never experienced this StackOverflowError myself. It this does solve the problem, I'll contribute a patch upstream to gitblit.
Thank you for the update. I just tried and get a Server Error when using it on the same repository. At least the previous one it shown the error but it shown the data in the GUI, this one shows "Server Error":
[2016-10-25 08:16:06,409] [HTTP-37] WARN org.eclipse.jetty.servlet.ServletHandler : Error for /plugins/gitblit/summary/;jsessionid=eqck15d7r2h6314pa636or8h java.lang.NoSuchMethodError: com.gitblit.models.RefModel.getReferencedObjectType()I at com.gitblit.wicket.panels.TagsPanel$1.populateItem(TagsPanel.java:78) at org.apache.wicket.markup.repeater.RefreshingView$1.newItem(RefreshingView.java:114) at org.apache.wicket.markup.repeater.DefaultItemReuseStrategy$1.next(DefaultItemReuseStrategy.java:71) at org.apache.wicket.markup.repeater.DefaultItemReuseStrategy$1.next(DefaultItemReuseStrategy.java:68) at org.apache.wicket.markup.repeater.RefreshingView.addItems(RefreshingView.java:189) at org.apache.wicket.markup.repeater.RefreshingView.onPopulate(RefreshingView.java:98) at org.apache.wicket.markup.repeater.AbstractRepeater.onBeforeRender(AbstractRepeater.java:131) at org.apache.wicket.markup.repeater.AbstractPageableView.onBeforeRender(AbstractPageableView.java:121) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1777) at org.apache.wicket.Component.onBeforeRender(Component.java:4001) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1777) at org.apache.wicket.Component.onBeforeRender(Component.java:4001) at org.apache.wicket.Page.onBeforeRender(Page.java:1555) at com.gitblit.wicket.pages.BasePage.onBeforeRender(BasePage.java:154) at com.gitblit.wicket.pages.RepositoryPage.onBeforeRender(RepositoryPage.java:680) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.Component.prepareForRender(Component.java:2297) at org.apache.wicket.Page.prepareForRender(Page.java:1545) at org.apache.wicket.Component.prepareForRender(Component.java:2329) at org.apache.wicket.Page.renderPage(Page.java:912) at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.respond(BookmarkablePageRequestTarget.java:261) at org.apache.wicket.request.AbstractRequestCycleProcessor.respond(AbstractRequestCycleProcessor.java:105) at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1287) at org.apache.wicket.RequestCycle.step(RequestCycle.java:1358) at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1465) at org.apache.wicket.RequestCycle.request(RequestCycle.java:545) at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486) at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319) at com.googlesource.gerrit.plugins.gitblit.GerritWicketFilter.doFilter(GerritWicketFilter.java:112) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:236) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73) at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117) at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:200) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178) at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:132) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:138) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) [2016-10-25 08:16:06,410] [HTTP-37] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /plugins/gitblit/summary/;jsessionid=eqck15d7r2h6314pa636or8h?r=STB_SW java.lang.NoSuchMethodError: com.gitblit.models.RefModel.getReferencedObjectType()I at com.gitblit.wicket.panels.TagsPanel$1.populateItem(TagsPanel.java:78) at org.apache.wicket.markup.repeater.RefreshingView$1.newItem(RefreshingView.java:114) at org.apache.wicket.markup.repeater.DefaultItemReuseStrategy$1.next(DefaultItemReuseStrategy.java:71) at org.apache.wicket.markup.repeater.DefaultItemReuseStrategy$1.next(DefaultItemReuseStrategy.java:68) at org.apache.wicket.markup.repeater.RefreshingView.addItems(RefreshingView.java:189) at org.apache.wicket.markup.repeater.RefreshingView.onPopulate(RefreshingView.java:98) at org.apache.wicket.markup.repeater.AbstractRepeater.onBeforeRender(AbstractRepeater.java:131) at org.apache.wicket.markup.repeater.AbstractPageableView.onBeforeRender(AbstractPageableView.java:121) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1777) at org.apache.wicket.Component.onBeforeRender(Component.java:4001) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.MarkupContainer.onBeforeRenderChildren(MarkupContainer.java:1777) at org.apache.wicket.Component.onBeforeRender(Component.java:4001) at org.apache.wicket.Page.onBeforeRender(Page.java:1555) at com.gitblit.wicket.pages.BasePage.onBeforeRender(BasePage.java:154) at com.gitblit.wicket.pages.RepositoryPage.onBeforeRender(RepositoryPage.java:680) at org.apache.wicket.Component.internalBeforeRender(Component.java:1069) at org.apache.wicket.Component.beforeRender(Component.java:1103) at org.apache.wicket.Component.prepareForRender(Component.java:2297) at org.apache.wicket.Page.prepareForRender(Page.java:1545) at org.apache.wicket.Component.prepareForRender(Component.java:2329) at org.apache.wicket.Page.renderPage(Page.java:912) at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.respond(BookmarkablePageRequestTarget.java:261) at org.apache.wicket.request.AbstractRequestCycleProcessor.respond(AbstractRequestCycleProcessor.java:105) at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1287) at org.apache.wicket.RequestCycle.step(RequestCycle.java:1358) at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1465) at org.apache.wicket.RequestCycle.request(RequestCycle.java:545) at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486) at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319) at com.googlesource.gerrit.plugins.gitblit.GerritWicketFilter.doFilter(GerritWicketFilter.java:112) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:236) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73) at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117) at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:200) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178) at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:65) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:132) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:138) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745)
What an embarrassing and stupid bug of my part. It's a typo. Don't know how that slipped in, and why I didn't see it in my tests. :-( Don't have time right now, but I'll fix that tonight.
Should be fixed now. If you re-download v2.12.171.4 again and install it in Gerrit, this NoSuchMethodError should no longer occur. Note that you have to re-download; I overwrote the previous broken v2.12.171.4.
Downloaded and tried. Same error than the original one. Maybe some of this from the gerrit.config can be of any help :
[container]
user = gerrit2
javaHome = /local/java/jdk1.7.0_67/
heapLimit = 32g
#javaOptions = -Dcom.sun.management.jmxremote.port=49999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
javaOptions = -Dcom.sun.management.jmxremote.port=49999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Djava.awt.headless=true
[cache]
directory = cache
[cache "diff"]
memoryLimit = 512m
timeout = 15s
[core]
packedGitOpenFiles = 1024mo
Also the gitblit.properties used is as follows:
#
#GitBlit custom settings
#
# Disable GitBlit Git Server, as Gerrit already serves Git repositories
git.cacheRepositoryList = false
git.enableGitServlet = false
git.onlyAccessBareRepositories = true
# Points to Gerrit Git repositories folder
git.repositoriesFolder = /git/gpk-apl-grt5a
# Disable cache repository list
git.cacheRepositoryList = false
# Disable GitBlit's SSH daemon. Gerrit has one already.
git.sshPort = 0
# The ticket service would need the git servlet. Anyway, this is functionality
# provided by Gerrit.
tickets.service = com.googlesource.gerrit.plugins.gitblit.app.ReallyNullTicketService
# Disable the fanout service
fanout.port = 0
# Allow the user of GitBlit cookie authentication for avoiding overload on
# Gerrit authentication every time that a repository is browsed
web.allowCookieAuthentication = true
# Requires users to authenticate against Gerrit before giving any access
web.authenticateViewPages = true
# Disable GitBlit Web administration, as it supposed to be made on Gerrit init
web.allowAdministration = false
web.authenticateAdminPages = true
# Disable the ability to fork a repository
web.allowForking = false
# Disable GitBlit RPC services
web.enableRpcServlet = false
web.enableRpcManagement = false
web.enableRpcAdministration = false
# No Commit Cache (default 14)
web.activityCacheDays = 0
# Disable the 'mount' of GitBlit parameters in the URL, it would mess-up the
# Gerrit plugins URLs generation and parsing of them when Gerrit repos have '/'
# in their name. Additionally avoid the usage of URL-encoded '/' (%2F) as it may
# be automatically decoded by reverse proxies and then even forbidden for security
# reasons.
web.mountParameters = false
web.allowZipDownloads = true
web.compressedDownloads = zip gz
Same error than the original one.
In other words, StackOverflowError?
Yes
The project has 15262 changes/patch sets.
With my change no JGit RevCommits should be serialized anymore. Either it's not that, or I missed some somewhere... I'll take another look.
Don't see anything. I'm out of ideas. Note that this StackOverflowError has also been reported against plain Gitblit: gitblit/gitblit#1011, and over at repo-discuss also against the official plugin.
Also Gerrit bug 3316 and this comment. Also this really old report against Gitblit 1.3.2/1.4.0.
I can try to add more logging to try to figure out where this is coming from...
Will help in any way raising the level of Gerrit logging ? But at least it trows the exception but does the job.
Will help in any way raising the level of Gerrit logging ?
Might be worth a try. But frankly said, I don't know where Gitblit/log4j picks up the logging config when run as a Gerrit plugin.
After some more research, I think I've found another problem spot: Gitblit uses in some cases anonymous inner classes that get serialized. That will also serialize some synthetic fields created to access final variables from outside. Such is the case for instance in HistoryPanel, which I thought I had fixed.
Alas, the Wicket DataView used (line 155) will contain a synthetic field to the final variable allRefs
(line 135). That variable is a map containing commits again (all branches with their target commits, keyed by target commit).
Not sure I can fix this in all cases in the plugin; depends on where else this code pattern occurs. The few cases directly related to JGit objects that I've found so far I can deal with; but I don't know what I'll find in other page/panel classes. But in any case I'll give it another try.
BTW, serializable inner classes are discouraged and what I see in HistoryPanel looks very similar to well-known anti-patterns for Wicket applications. :-(
Let's try again:
So if my conjecture that the StackOverflowError comes from inadvertently serializing JGit RevCommit objects with long parent chains is correct, we stand a fair chance that v2.12.171.5 doesn't have that bug anymore.
Could you try again, please?
I pulled the v2.12.171.4 release. It had already been downloaded 6 times, so I didn't want to overwrite it again.
Great Tomas. I think you are right in the new findings. It is working OK now. Maybe you should spread this change as well for the 2.13 plugin. And maybe let the original GitBlit know about it.
Plugin release v2.13.171.1 is published.
Pull request for upstream Gitblit is gitblit/gitblit#1141.
Thanks a lot for testing!
Gerrit 2.12.2 with GitBlit 2.12.162.1
We regularly see Gerrit's
error_log
flooded with