Connections possibly left unclosed

[vrtareg]

Hi Team

We are using this library in one of the Jira PlugIns to connect to the Gerrit server to gather the reviwes information.

It looks like that in our environment library leaves some unclosed leftover connections which then affecting PlugIn functionality as server denies connection because of conn limit from Gerrit server side.

Is it possible that somewhere exception is not catched and connection left open?

2016-11-21 14:18:58,526 Caesium-1-4 DEBUG ServiceRunner [c.m.p.j.gerrit.multipleRepo.MultipleRepoConfigManagerImpl] ssh connection failed com.sonymobile.tools.gerrit.gerritevents.ssh.SshException: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 7 Too many concurrent connections at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.connect(SshConnectionImpl.java:181) at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:98) at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:77) at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:59) at com.meetme.plugins.jira.gerrit.multipleRepo.MultipleRepoConfigManagerImpl.createBareClone(MultipleRepoConfigManagerImpl.java:538) at com.meetme.plugins.jira.gerrit.multipleRepo.MultipleRepoConfigManagerImpl.synchronize(MultipleRepoConfigManagerImpl.java:622) at com.meetme.plugins.jira.gerrit.index.RevisionIndexer.updateIndexes(RevisionIndexer.java:223) at com.meetme.plugins.jira.gerrit.scheduling.SynchronizerLauncher.execute(SynchronizerLauncher.java:36) at com.atlassian.sal.jira.scheduling.JiraPluginScheduler$JobDescriptor.runJob(JiraPluginScheduler.java:110) at com.atlassian.sal.jira.scheduling.JiraPluginScheduler.runJob(JiraPluginScheduler.java:80) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:410) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:388) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285) at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59) at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) at java.lang.Thread.run(Thread.java:745) Caused by: com.jcraft.jsch.JSchException: SSH_MSG_DISCONNECT: 7 Too many concurrent connections at com.jcraft.jsch.Session.read(Session.java:987) at com.jcraft.jsch.UserAuthPublicKey.start(UserAuthPublicKey.java:198) at com.jcraft.jsch.Session.connect(Session.java:463) at com.jcraft.jsch.Session.connect(Session.java:183) at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.connect(SshConnectionImpl.java:177) ... 21 more

Regards, Areg

[rsandell]

That could be possible, the connection is designed to try to keep the stream flowing as uninterrupted as possible. So if it looses connection somehow it tries to reconnect. If something lingers on on either end it would build up.

[vrtareg]

Any way to prevent this and close unused connections?

[rsandell]

You would need to find it first :) then it would hopefully be pretty obvious how to fix the leak if you find where it is.

[vrtareg]

Any recommendations what to monitor and where to put debug output? I will be glad to do if I know how.

[rsandell]

The connect logic is the last line of the stack trace you provided. The main loop is here https://github.com/sonyxperiadev/gerrit-events/blob/master/src/main/java/com/sonymobile/tools/gerrit/gerritevents/GerritConnection.java#L314

[vrtareg]

So something is going wrong and new connection is created without closing current one. This is really killing as Gerrit servers has a limit on incoming connections from clients and on a next round it just denies new connection. Only way to fix this is to restart Jira instance ...

[rsandell]

Which is strange because if a connection fails to be established there shouldn't be any connection available to close.