sonyxperiadev / gerrit-events

MIT License
47 stars 62 forks source link

Gerrit Events not causing a trigger #71

Open thadguidry opened 7 years ago

thadguidry commented 7 years ago

We don't see the issue in #70 any longer, and we're not sure why, perhaps networking changes we are unaware of... We have downgraded to Gerrit Trigger 2.23.3 But Events are still not triggering a Jenkins pipeline job consistently, the trigger typically stops during the night and we are having to restart the Jenkins server the next morning to get triggers working again.

We captured a heap dump this morning just before we did a Jenkins restart to get the triggers happening again on this same server. After the restart, triggers began to happen again. Please analyze and let us know how we can help further.

I see this potential thread as suspect ???

"Connect thread gerrit.ericsson.se session" prio=5 tid=91 TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
    at java.io.PipedInputStream.receive(PipedInputStream.java:231)
    at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
    at com.jcraft.jsch.Channel$PassiveOutputStream.write(Channel.java:660)
    at com.jcraft.jsch.IO.put(IO.java:64)
       Local Variable: com.jcraft.jsch.IO#1
    at com.jcraft.jsch.Channel.write(Channel.java:438)
    at com.jcraft.jsch.Session.run(Session.java:1448)
       Local Variable: int[]#9794
       Local Variable: com.jcraft.jsch.Buffer#2
       Local Variable: com.jcraft.jsch.Packet#2
       Local Variable: int[]#9793
       Local Variable: byte[]#32460
       Local Variable: com.jcraft.jsch.Session#1
    at java.lang.Thread.run(Thread.java:745)

Heap Dump: https://drive.google.com/file/d/0B533WzlrxWraWTZaOElXSGN5M1U/view?usp=sharing

azjps commented 6 years ago

+1 Also encountering the same issue sporadically (maybe once a month). Also on Gerrit Trigger 2.23.3 (and Jenkins 2.66). $JENKINS_URL/threadDump shows:

"Connect thread [$gerrit_server] session" Id=72561 Group=main TIMED_WAITING on com.jcraft.jsch.Channel$MyPipedInputStream@3a91362d
    at java.lang.Object.wait(Native Method)
    -  waiting on com.jcraft.jsch.Channel$MyPipedInputStream@3a91362d
    at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
    at java.io.PipedInputStream.receive(PipedInputStream.java:231)
    at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
    at com.jcraft.jsch.Channel$PassiveOutputStream.write(Channel.java:660)
    at com.jcraft.jsch.IO.put(IO.java:64)
    at com.jcraft.jsch.Channel.write(Channel.java:438)
    at com.jcraft.jsch.Session.run(Session.java:1448)
    at java.lang.Thread.run(Thread.java:745)

To get around it without a full Jenkins restart, I have just been going to Manage Jenkins > Gerrit Trigger, toggling the Status of the Gerrit server connection to off and then back on. (And then re-triggering the missed jobs via Query and Trigger Gerrit Patches with some complicated label query.)