Open thadguidry opened 7 years ago
+1 Also encountering the same issue sporadically (maybe once a month). Also on Gerrit Trigger 2.23.3 (and Jenkins 2.66). $JENKINS_URL/threadDump
shows:
"Connect thread [$gerrit_server] session" Id=72561 Group=main TIMED_WAITING on com.jcraft.jsch.Channel$MyPipedInputStream@3a91362d
at java.lang.Object.wait(Native Method)
- waiting on com.jcraft.jsch.Channel$MyPipedInputStream@3a91362d
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
at java.io.PipedInputStream.receive(PipedInputStream.java:231)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at com.jcraft.jsch.Channel$PassiveOutputStream.write(Channel.java:660)
at com.jcraft.jsch.IO.put(IO.java:64)
at com.jcraft.jsch.Channel.write(Channel.java:438)
at com.jcraft.jsch.Session.run(Session.java:1448)
at java.lang.Thread.run(Thread.java:745)
To get around it without a full Jenkins restart, I have just been going to Manage Jenkins > Gerrit Trigger
, toggling the Status
of the Gerrit server connection to off and then back on. (And then re-triggering the missed jobs via Query and Trigger Gerrit Patches
with some complicated label query.)
We don't see the issue in #70 any longer, and we're not sure why, perhaps networking changes we are unaware of... We have downgraded to Gerrit Trigger 2.23.3 But Events are still not triggering a Jenkins pipeline job consistently, the trigger typically stops during the night and we are having to restart the Jenkins server the next morning to get triggers working again.
We captured a heap dump this morning just before we did a Jenkins restart to get the triggers happening again on this same server. After the restart, triggers began to happen again. Please analyze and let us know how we can help further.
I see this potential thread as suspect ???
Heap Dump: https://drive.google.com/file/d/0B533WzlrxWraWTZaOElXSGN5M1U/view?usp=sharing