search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-14362] 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException #2407

Closed timja closed 11 years ago

timja commented 12 years ago

Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time:

I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this.

This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below.

Originally reported by gcc, imported from: 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException
  • assignee: lmcazra
  • status: Resolved
  • priority: Critical
  • resolution: Fixed
  • resolved: 2013-06-19T12:41:09+00:00
  • imported: 2022/01/10
timja commented 12 years ago

davidparsson:

I think I have seen this issue (or a very similar one) when trying to generate Performance reports for huge JMeter reports. Jenkins v1.471 and Performance plugin v1.8.

timja commented 11 years ago

bob_lloyd:

I have the same issue, but cannot track it to a specific job. I am not generating Performance reports (to my knowledge, unless it's happening w/o my intention). I've attached my log below (though, it's pretty much the same as above). I'm running with Jenkins 1.491 with Sun JDK 1.6.0_26

This happens for me after about 36 hours of up-time for Jenkins. I have ~30 jobs running on 6 servers. One server runs jobs pretty much constantly, while other servers run much less frequently.

"RequestHandlerThread541" daemon prio=6 tid=0x4959d800 nid=0x8c4 runnable [0x4b40f000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Unknown Source)

Locked ownable synchronizers:

timja commented 11 years ago

josesa:

I still have this problem occurring at least twice a week causing cpu at 400% (on a 4 CPUs machine) that forces me to restart the server.

When it happens I can workaround it (temporarily) by using the 'monitoring' plugin, going to the list of threads and sort the table by execution time and kill the threads actively running "DeflaterOutputStream.deflate" that are on top of the table. This makes CPU drop back to normal values but eventually a restart will be needed at the end of the day.

Here is a stack trace collected from our logs:

WARNING: Untrapped servlet exception
winstone.ClientSocketException: Failed to write to client
at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:181)
at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:119)
at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:169)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238)
at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
at net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:46)
at java.io.FilterOutputStream.close(FilterOutputStream.java:160)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
at java.io.BufferedWriter.close(BufferedWriter.java:266)
at org.dom4j.io.XMLWriter.close(XMLWriter.java:286)
at org.kohsuke.stapler.jelly.HTMLWriterOutput.close(HTMLWriterOutput.java:70)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:56)
at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107)
at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488)
at org.kohsuke.stapler.Stapler.service(Stapler.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
... 88 more
timja commented 11 years ago

bedge:

I have the same problem. It triggers soft lockups in the kernel of the jenkins host:

901 Jan 30 02:31:58 build kernel: [589969.228931] INFO: rcu_sched detected stall on CPU 3 (t=57719 jiffies)
902 Jan 30 02:31:58 build kernel: [589969.228939] INFO: rcu_sched detected stall on CPU 2 (t=57719 jiffies)
903 Jan 30 02:31:58 build kernel: [589969.228929] INFO: rcu_sched detected stall on CPU 1 (t=57719 jiffies)
904 Jan 30 02:31:58 build kernel: [589969.228944] INFO: rcu_sched detected stall on CPU 4 (t=57719 jiffies)
905 Jan 30 02:31:58 build kernel: [589969.228939] sending NMI to all CPUs:

Note the timestamp correlation between the exception below and the stalls above.

712 Jan 30, 2013 2:31:58 AM org.kohsuke.stapler.compression.CompressionFilter reportException
6713 WARNING: Untrapped servlet exception
6714 winstone.ClientSocketException: Failed to write to client
6715 at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
6716 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
6717 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
6718 at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:165)
6719 at winstone.WinstoneOutputStream.flush(WinstoneOutputStream.java:217)
6720 at winstone.WinstoneOutputStream.close(WinstoneOutputStream.java:227)
6721 at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241)
6722 at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
6723 at net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:46)

This happens most nights at the same time.

timja commented 11 years ago

ricko:

On my system, just upgraded to 1.492 from the last LTS 1.480.1, this happens whenever I display an EMMA code coverage report.
The server java process takes upwards of 80% CPU for at least 20 min until I use the method above to kill the "Active Request" in the Monitoring plugin.
I've seen this behavior for months, but this jira thread shows me how to kill the thread. That's progress.

timja commented 11 years ago

hoppej:

Hi,

we have been running into these troubles often the last time.
I found a "disable" flag for "reportException".
Add -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true to your VM-Options in jenkins xml!

Good luck,
Jan

timja commented 11 years ago

josesa:

I've added that entry to my startup script 2 weeks ago and also haven't experienced 100% cpu symptom since then.

timja commented 11 years ago

ustuehler:

Could this be related to the thread-safety (or not) of java.util.zip?

https://forums.oracle.com/forums/thread.jspa?messageID=4627097

In GNU Classpath, java.util.zip.Deflater is clearly documented as not thread-safe due to limitations of the API. I couldn't find a mention of the thread-safety issue in Sun/Oracle documentation, but some have concluded that their implementation isn't thread-safe either.

timja commented 11 years ago

rborer:

Hi,

I'm also experiencing this issue with Jenkins version 1.506 on the following environment:

timja commented 11 years ago

sinimakela:

We are having this same issue with Jenkins v1.510.

The workaround suggested from Jan Hoppe seem to have solved the problem also in our environment. After adding "JAVA_ARGS=-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" into /etc/default/jenkins the CPU usage has normalized based on a couple of days experience.

timja commented 11 years ago

hoppej:

I blogged a little bit about detecting that problem: http://devophuman.blogspot.com/2013/04/jenkins-is-going-wild.html

timja commented 11 years ago

david_catalan:

Hello, we are experiencing the same issue on our 2 Jenkins instances. I have found a way to reproduce the issue:
1. Create a job which generates a big console output (minimum 25Mb) (ie. a shell action doing a cat of a big text file)
2. On your screen, keep an eye on the jenkins log (file tail) + the cpu utilization of the jenkins process (top command or visualVM)
3. On Jenkins interface, navigate to the full console output of this job (http:///job/ 4. Keep refreshing the console page without letting the page load completly
=> You should see the "java.util.zip.Deflater.deflate" stack trace in the the jenkins.log
5. Keep refreshing until the Jenkins process uses all the CPU(s)!!

Tested on our 2 instances and, depending on the box current use + the size of the console, the CPU issue takes more or less time to occur.

Finally, after applying the "-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" fix, the exception still occurs in the log BUT, no more CPU overload!!

I see that a lot of users are experiencing this issue on multiple configuration (OS/Jenkins Version)!!
Also a lot of users are spending time on it trying to debug and find out the reason/solution of this issue.

Now, I would like to see some reaction from the Jenkins team. Are they taking care of this issue? We expect some solution from their side.

Thanks in advance,
David.

timja commented 11 years ago

walterk82:

I have tried option -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true however, the Content-Encoding header that is returned is always of type gzip. Does it matter where in the arguments section that this needs to be set?

After looking a little more at the code it seems that the disable doesn't actually disable to compression just the error reporting? Why would this be?

timja commented 11 years ago

ostersc:

I have this problem as well with jenkins version 1.509. Using the JavaMelody dashboard to kill the threads stuck in ava.util.zip.DeflaterOutputStream.deflate reduced the CPU usage (without seeming to cause problems).

timja commented 11 years ago

coreyoconnor:

We are using jenkins 1.484 and are experiencing the same issue. The "-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" option did result in the threads no longer taking 100% CPU. However, this broke loading the CSS. Perhaps due to what Walter Kacynski noticed.

timja commented 11 years ago

guykisel:

On 1.511 this is still happening, even though I set:

-Xrs -Xmx1024m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8081 --prefix=/jenkins -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true

timja commented 11 years ago

coreyoconnor:

Since my previous comment I have noticed two incidents of 100% cpu usage from a zip compression thread that appears "stuck". Either org.kohsuke.stapler.compression.CompressionFilter.disabled=true reduces the frequency ; a different path results in the same behavior ; or the reduction in frequency is all by chance.

timja commented 11 years ago

hoppej:

You should create a thread dump from your vm to enrich this report. Configuration is mentioned in http://devophuman.blogspot.com/2013/04/jenkins-is-going-wild.html. The VisualVM is part of Oracles JDK...

timja commented 11 years ago

jaens:

Experiencing this daily as well. Fairly frustrating.

Jenkins 1.514

java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Stack trace of looping thread:

  java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:322)
- locked <0x000000008cbec4d8> (a java.util.zip.ZStreamRef)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:176)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:135)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:89)
- locked <0x000000008cbec490> (a java.util.zip.GZIPOutputStream)
at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:315)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:148)
- locked <0x000000008cc1a478> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
at java.io.PrintWriter.close(PrintWriter.java:312)
- locked <0x000000008cc1a478> (a java.io.OutputStreamWriter)
at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
timja commented 11 years ago

guykisel:

Thread dump of this issue occuring on 1.516:

Thread Dump

AWT-Windows

"AWT-Windows" Id=217 Group=main RUNNABLE (in native)
    at sun.awt.windows.WToolkit.eventLoop(Native Method)
    at sun.awt.windows.WToolkit.run(WToolkit.java:299)
    at java.lang.Thread.run(Thread.java:722)

bot1-input

"bot1-input" Id=167 Group=main RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    -  locked java.lang.Object@362d4a04
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    -  locked sun.security.ssl.AppInputStream@57566865
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    -  locked java.io.InputStreamReader@69dfebae
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
    -  locked java.io.InputStreamReader@69dfebae
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at org.pircbotx.InputThread.run(InputThread.java:83)

bot1-output

"bot1-output" Id=168 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.pircbotx.OutputThread.run(OutputThread.java:101)

Channel reader thread: serial-server-1

"Channel reader thread: serial-server-1" Id=146 Group=main WAITING on java.lang.Object@61e52512
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.Object@61e52512
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144)
    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272)
    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565)
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Channel reader thread: ubuntu-test-VM-3

"Channel reader thread: ubuntu-test-VM-3" Id=148 Group=main WAITING on java.lang.Object@1c5e0a0a
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.Object@1c5e0a0a
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144)
    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272)
    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565)
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

ConnectorThread:[http-8081]

"ConnectorThread:[http-8081]" Id=12 Group=main RUNNABLE (in native)
    at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:125)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
    -  locked java.net.SocksSocketImpl@773d4576
    at java.net.ServerSocket.implAccept(ServerSocket.java:522)
    at java.net.ServerSocket.accept(ServerSocket.java:490)
    at winstone.HttpListener.run(HttpListener.java:139)
    at java.lang.Thread.run(Thread.java:722)

DestroyJavaVM

"DestroyJavaVM" Id=15 Group=main RUNNABLE

Executor #0 for D0-Debian-Slave-1

"Executor #0 for D0-Debian-Slave-1" Id=86 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D0-Debian-Slave-2

"Executor #0 for D0-Debian-Slave-2" Id=91 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D1-debian-FreshInstall-1

"Executor #0 for D1-debian-FreshInstall-1" Id=116 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D2-Debian-Slave-1

"Executor #0 for D2-Debian-Slave-1" Id=111 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D2-Debian-Slave-2

"Executor #0 for D2-Debian-Slave-2" Id=106 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D3-Debian-Slave-1

"Executor #0 for D3-Debian-Slave-1" Id=96 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for D3-Debian-Slave-2

"Executor #0 for D3-Debian-Slave-2" Id=101 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for master

"Executor #0 for master" Id=38 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for serial-server-1

"Executor #0 for serial-server-1" Id=48 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for ubuntu-test-VM-1

"Executor #0 for ubuntu-test-VM-1" Id=64 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for ubuntu-test-VM-2

"Executor #0 for ubuntu-test-VM-2" Id=75 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #0 for ubuntu-test-VM-3

"Executor #0 for ubuntu-test-VM-3" Id=53 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D0-Debian-Slave-1

"Executor #1 for D0-Debian-Slave-1" Id=87 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D0-Debian-Slave-2

"Executor #1 for D0-Debian-Slave-2" Id=92 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D2-Debian-Slave-1

"Executor #1 for D2-Debian-Slave-1" Id=112 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D2-Debian-Slave-2

"Executor #1 for D2-Debian-Slave-2" Id=107 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D3-Debian-Slave-1

"Executor #1 for D3-Debian-Slave-1" Id=97 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for D3-Debian-Slave-2

"Executor #1 for D3-Debian-Slave-2" Id=102 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for master

"Executor #1 for master" Id=39 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for serial-server-1

"Executor #1 for serial-server-1" Id=49 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for ubuntu-test-VM-1

"Executor #1 for ubuntu-test-VM-1" Id=65 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for ubuntu-test-VM-2

"Executor #1 for ubuntu-test-VM-2" Id=76 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #1 for ubuntu-test-VM-3

"Executor #1 for ubuntu-test-VM-3" Id=54 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D0-Debian-Slave-1

"Executor #2 for D0-Debian-Slave-1" Id=88 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D0-Debian-Slave-2

"Executor #2 for D0-Debian-Slave-2" Id=93 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D2-Debian-Slave-1

"Executor #2 for D2-Debian-Slave-1" Id=113 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D2-Debian-Slave-2

"Executor #2 for D2-Debian-Slave-2" Id=108 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D3-Debian-Slave-1

"Executor #2 for D3-Debian-Slave-1" Id=98 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for D3-Debian-Slave-2

"Executor #2 for D3-Debian-Slave-2" Id=103 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for master

"Executor #2 for master" Id=40 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for serial-server-1

"Executor #2 for serial-server-1" Id=50 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for ubuntu-test-VM-1

"Executor #2 for ubuntu-test-VM-1" Id=66 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for ubuntu-test-VM-2

"Executor #2 for ubuntu-test-VM-2" Id=77 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #2 for ubuntu-test-VM-3

"Executor #2 for ubuntu-test-VM-3" Id=55 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D0-Debian-Slave-1

"Executor #3 for D0-Debian-Slave-1" Id=89 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D0-Debian-Slave-2

"Executor #3 for D0-Debian-Slave-2" Id=94 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D2-Debian-Slave-1

"Executor #3 for D2-Debian-Slave-1" Id=114 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D2-Debian-Slave-2

"Executor #3 for D2-Debian-Slave-2" Id=109 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D3-Debian-Slave-1

"Executor #3 for D3-Debian-Slave-1" Id=99 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for D3-Debian-Slave-2

"Executor #3 for D3-Debian-Slave-2" Id=104 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for master

"Executor #3 for master" Id=41 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for serial-server-1

"Executor #3 for serial-server-1" Id=51 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for ubuntu-test-VM-1

"Executor #3 for ubuntu-test-VM-1" Id=67 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for ubuntu-test-VM-2

"Executor #3 for ubuntu-test-VM-2" Id=78 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #3 for ubuntu-test-VM-3

"Executor #3 for ubuntu-test-VM-3" Id=56 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D0-Debian-Slave-1

"Executor #4 for D0-Debian-Slave-1" Id=90 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D0-Debian-Slave-2

"Executor #4 for D0-Debian-Slave-2" Id=95 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D2-Debian-Slave-1

"Executor #4 for D2-Debian-Slave-1" Id=115 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D2-Debian-Slave-2

"Executor #4 for D2-Debian-Slave-2" Id=110 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D3-Debian-Slave-1

"Executor #4 for D3-Debian-Slave-1" Id=100 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for D3-Debian-Slave-2

"Executor #4 for D3-Debian-Slave-2" Id=105 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for master

"Executor #4 for master" Id=42 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for ubuntu-test-VM-1

"Executor #4 for ubuntu-test-VM-1" Id=68 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for ubuntu-test-VM-2

"Executor #4 for ubuntu-test-VM-2" Id=79 Group=main TIMED_WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at hudson.util.OneShotEvent.block(OneShotEvent.java:86)
    at hudson.model.Queue.pop(Queue.java:884)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #4 for ubuntu-test-VM-3

"Executor #4 for ubuntu-test-VM-3" Id=57 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #5 for master

"Executor #5 for master" Id=43 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #5 for ubuntu-test-VM-1

"Executor #5 for ubuntu-test-VM-1" Id=69 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #5 for ubuntu-test-VM-2

"Executor #5 for ubuntu-test-VM-2" Id=80 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #5 for ubuntu-test-VM-3

"Executor #5 for ubuntu-test-VM-3" Id=58 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #6 for master

"Executor #6 for master" Id=44 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #6 for ubuntu-test-VM-1

"Executor #6 for ubuntu-test-VM-1" Id=70 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #6 for ubuntu-test-VM-2

"Executor #6 for ubuntu-test-VM-2" Id=81 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #6 for ubuntu-test-VM-3

"Executor #6 for ubuntu-test-VM-3" Id=59 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #7 for master

"Executor #7 for master" Id=45 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #7 for ubuntu-test-VM-1

"Executor #7 for ubuntu-test-VM-1" Id=71 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #7 for ubuntu-test-VM-2

"Executor #7 for ubuntu-test-VM-2" Id=82 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #7 for ubuntu-test-VM-3

"Executor #7 for ubuntu-test-VM-3" Id=60 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #8 for master

"Executor #8 for master" Id=46 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #8 for ubuntu-test-VM-1

"Executor #8 for ubuntu-test-VM-1" Id=72 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #8 for ubuntu-test-VM-2

"Executor #8 for ubuntu-test-VM-2" Id=83 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #8 for ubuntu-test-VM-3

"Executor #8 for ubuntu-test-VM-3" Id=61 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #9 for master

"Executor #9 for master" Id=47 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #9 for ubuntu-test-VM-1

"Executor #9 for ubuntu-test-VM-1" Id=73 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #9 for ubuntu-test-VM-2

"Executor #9 for ubuntu-test-VM-2" Id=84 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Executor #9 for ubuntu-test-VM-3

"Executor #9 for ubuntu-test-VM-3" Id=62 Group=main WAITING on hudson.model.Queue@156b238
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.model.Queue@156b238
    at java.lang.Object.wait(Object.java:503)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    at hudson.model.Queue.pop(Queue.java:882)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)

Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2]

"Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2]" Id=25 Group=main RUNNABLE
 at java.lang.String.intern(Native Method)
    at java.lang.Class.searchMethods(Class.java:2670)
    at java.lang.Class.getMethod0(Class.java:2694)
    at java.lang.Class.getMethod(Class.java:1622)
    at org.apache.commons.logging.LogFactory.directGetContextClassLoader(LogFactory.java:896)
    at org.apache.commons.logging.LogFactory$1.run(LogFactory.java:862)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.commons.logging.LogFactory.getContextClassLoaderInternal(LogFactory.java:859)
    at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:423)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
    at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:942)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:274)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:220)
    at org.apache.commons.digester.Rule.end(Rule.java:257)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1345)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1742)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2900)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
    at org.apache.commons.digester.Digester.parse(Digester.java:1871)
    at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76)
    at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43)
    at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:922)
    at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:896)
    at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:461)
    at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1581)
    at hudson.model.User.getProjects(User.java:449)
    at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21)
    at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:112)
    at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:532)
    at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
    at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
    at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
    at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
    at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
    at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
    at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
    at org.kohsuke.stapler.export.Model.writeTo(Model.java:154)
    at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:264)
    at hudson.model.Api.doJson(Api.java:193)
    at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
    at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@128d38f3

Handling GET /jenkins/threadDump : RequestHandlerThread[#7]

"Handling GET /jenkins/threadDump : RequestHandlerThread[#7]" Id=33 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@81e7b1d
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.FutureTask$Sync@81e7b1d
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
    at java.util.concurrent.FutureTask.get(FutureTask.java:119)
    at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81)
    at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2865)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
    at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
    at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
    at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
    at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
    at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
    at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
    at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
    at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
    at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@567bd2f9

IM-Reconnector-Thread

"IM-Reconnector-Thread" Id=21 Group=main WAITING on java.util.concurrent.Semaphore$NonfairSync@4c423981
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.Semaphore$NonfairSync@4c423981
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
    at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:160)
    at java.lang.Thread.run(Thread.java:722)

javamelody jenkins

"javamelody jenkins" Id=23 Group=main TIMED_WAITING on java.util.TaskQueue@5062da8
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@5062da8
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

Jenkins cron thread

"Jenkins cron thread" Id=16 Group=main TIMED_WAITING on java.util.TaskQueue@fcd6b44
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@fcd6b44
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

Jenkins UDP 33848 monitoring thread

"Jenkins UDP 33848 monitoring thread" Id=36 Group=main RUNNABLE (in native)
    at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
    -  locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37
    at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90)
    -  locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37
    at java.net.DatagramSocket.receive(DatagramSocket.java:786)
    -  locked java.net.DatagramPacket@4945a2e3
    -  locked java.net.MulticastSocket@7ac90aa9
    at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82)

JmDNS(FITBPWEB.local.).State.Timer

"JmDNS(FITBPWEB.local.).State.Timer" Id=126 Group=main TIMED_WAITING on java.util.TaskQueue@187672e7
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@187672e7
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

JmDNS(FITBPWEB.local.).Timer

"JmDNS(FITBPWEB.local.).Timer" Id=125 Group=main TIMED_WAITING on java.util.TaskQueue@871a83d
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@871a83d
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

jrobin jenkins

"jrobin  jenkins" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@57f4be7d
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.TaskQueue@57f4be7d
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

LauncherControlThread[ControlPort=-1]

"LauncherControlThread[ControlPort=-1]" Id=13 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at winstone.Launcher.run(Launcher.java:265)
    at java.lang.Thread.run(Thread.java:722)

NioSocketAcceptor-1

"NioSocketAcceptor-1" Id=34 Group=main RUNNABLE (in native)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    -  locked sun.nio.ch.Util$2@3298e7c
    -  locked java.util.Collections$UnmodifiableSet@43662b65
    -  locked sun.nio.ch.WindowsSelectorImpl@5ee2fc3e
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
    at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
    at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6ea937ab

Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1

"Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1" Id=150 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at hudson.remoting.PingThread.run(PingThread.java:86)

Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3

"Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3" Id=158 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at hudson.remoting.PingThread.run(PingThread.java:86)

pool-1-svnkit-thread-1

"pool-1-svnkit-thread-1" Id=149 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-11-thread-1

"pool-11-thread-1" Id=159 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-15-thread-1

"pool-15-thread-1" Id=118 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-3-thread-12

"pool-3-thread-12" Id=222 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.jenkinsci.plugins.vSphereCloudLauncher.launch(vSphereCloudLauncher.java:197)
    at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6de7a4d

pool-3-thread-16

"pool-3-thread-16" Id=238 Group=main RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
    at hudson.Functions.getThreadInfos(Functions.java:1061)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
    at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@32a5f72a

pool-4-thread-1

"pool-4-thread-1" Id=22 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-5-thread-1

"pool-5-thread-1" Id=235 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-7-thread-1

"pool-7-thread-1" Id=225 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-7-thread-2

"pool-7-thread-2" Id=226 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

pool-7-thread-3

"pool-7-thread-3" Id=241 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#14]

"RequestHandlerThread[#14]" Id=206 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#16]

"RequestHandlerThread[#16]" Id=208 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#20]

"RequestHandlerThread[#20]" Id=212 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#22]

"RequestHandlerThread[#22]" Id=214 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#23]

"RequestHandlerThread[#23]" Id=243 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#3]

"RequestHandlerThread[#3]" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

RequestHandlerThread[#6]

"RequestHandlerThread[#6]" Id=32 Group=main RUNNABLE
    at java.util.zip.Deflater.deflateBytes(Native Method)
    at java.util.zip.Deflater.deflate(Deflater.java:427)
    -  locked java.util.zip.ZStreamRef@924b115
    at java.util.zip.Deflater.deflate(Deflater.java:350)
    at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:251)
    at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
    at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146)
    -  locked java.util.zip.GZIPOutputStream@446b5a75
    at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
    -  locked java.io.OutputStreamWriter@325c9870
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
    at java.io.PrintWriter.close(PrintWriter.java:339)
    -  locked java.io.OutputStreamWriter@325c9870
    at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@51cbc1b5

SocketListener(FITBPWEB.local.)

"SocketListener(FITBPWEB.local.)" Id=124 Group=main RUNNABLE (in native)
    at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method)
    -  locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f
    at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90)
    -  locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f
    at java.net.DatagramSocket.receive(DatagramSocket.java:786)
    -  locked java.net.DatagramPacket@2f5cf3c8
    -  locked java.net.MulticastSocket@7f88e7c2
    at javax.jmdns.impl.SocketListener.run(SocketListener.java:41)

stderr copier for remote agent on serial-server-1

"stderr copier for remote agent on serial-server-1" Id=142 Group=main WAITING on java.lang.Object@3d0a76c2
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.Object@3d0a76c2
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171)
    at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)

stderr copier for remote agent on ubuntu-test-VM-3

"stderr copier for remote agent on ubuntu-test-VM-3" Id=145 Group=main WAITING on java.lang.Object@ae4f2f4
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.Object@ae4f2f4
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213)
    at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171)
    at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)

TCP slave agent listener port=0

"TCP slave agent listener port=0" Id=35 Group=main RUNNABLE (in native)
    at java.net.DualStackPlainSocketImpl.accept0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
    -  locked java.net.SocksSocketImpl@24db1c39
    at java.net.ServerSocket.implAccept(ServerSocket.java:522)
    at java.net.ServerSocket.accept(ServerSocket.java:490)
    at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91)

Thread-10

"Thread-10" Id=127 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

Thread-11

"Thread-11" Id=140 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc
    at java.lang.Object.wait(Native Method)
    -  waiting on com.trilead.ssh2.channel.Channel@79fad9dc
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210)
    at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70)
    at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48)

Thread-12

"Thread-12" Id=141 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc
    at java.lang.Object.wait(Native Method)
    -  waiting on com.trilead.ssh2.channel.Channel@79fad9dc
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210)
    at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70)
    at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48)

Thread-13

"Thread-13" Id=143 Group=main RUNNABLE
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210)
    at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70)
    at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48)

Thread-14

"Thread-14" Id=144 Group=main RUNNABLE
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210)
    at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
    at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70)
    at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48)

Thread-21

"Thread-21" Id=187 Group=main RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    -  locked java.io.BufferedInputStream@20d1ed39
    at com.sun.jndi.ldap.Connection.run(Connection.java:849)
    at java.lang.Thread.run(Thread.java:722)

Thread-6

"Thread-6" Id=120 Group=main RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
    at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
    at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677)
    at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475)
    at java.lang.Thread.run(Thread.java:722)

Thread-7

"Thread-7" Id=121 Group=main RUNNABLE (in native)
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
    at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
    at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
    at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677)
    at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475)
    at java.lang.Thread.run(Thread.java:722)

ViewJob reload thread

"ViewJob reload thread" Id=30 Group=main TIMED_WAITING on java.util.LinkedHashSet@5d1bfae4
    at java.lang.Object.wait(Native Method)
    -  waiting on java.util.LinkedHashSet@5d1bfae4
    at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:171)
    at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:188)

WinstoneHostConfigurationMgmt:default

"WinstoneHostConfigurationMgmt:default" Id=11 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at winstone.HostConfiguration.run(HostConfiguration.java:174)
    at java.lang.Thread.run(Thread.java:722)

Attach Listener

"Attach Listener" Id=4 Group=system RUNNABLE

Finalizer

"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@22be5a1d
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@22be5a1d
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

Java2D Disposer

"Java2D Disposer" Id=9 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@579b8322
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@579b8322
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at sun.java2d.Disposer.run(Disposer.java:145)
    at java.lang.Thread.run(Thread.java:722)

Reference Handler

"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7f0e37bc
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@7f0e37bc
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

D0-Debian-Slave-2

N/A

offline
D2-Debian-Slave-1

N/A

offline
D0-Debian-Slave-1

N/A

offline
D2-Debian-Slave-2

N/A

offline
serial-server-1

Channel reader thread: channel

"Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native)
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:242)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    -  locked java.io.BufferedInputStream@1348b49
    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266)
    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559)
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

main

"main" Id=1 Group=main WAITING on hudson.remoting.Channel@6a21b2
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.remoting.Channel@6a21b2
    at java.lang.Object.wait(Object.java:503)
    at hudson.remoting.Channel.join(Channel.java:800)
    at hudson.remoting.Launcher.main(Launcher.java:484)
    at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430)
    at hudson.remoting.Launcher.run(Launcher.java:221)
    at hudson.remoting.Launcher.main(Launcher.java:180)

Ping thread for channel hudson.remoting.Channel@6a21b2:channel

"Ping thread for channel hudson.remoting.Channel@6a21b2:channel" Id=10 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at hudson.remoting.PingThread.run(PingThread.java:86)

Pipe writer thread: channel

"Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

pool-1-thread-10

"pool-1-thread-10" Id=22 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

pool-1-thread-9

"pool-1-thread-9" Id=21 Group=main RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
    at hudson.Functions.getThreadInfos(Functions.java:1061)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@1a970

Finalizer

"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@a49259
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@a49259
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

Reference Handler

"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@842f23
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@842f23
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

Signal Dispatcher

"Signal Dispatcher" Id=4 Group=system RUNNABLE

D3-Debian-Slave-2

N/A

offline
ubuntu-test-VM-2

N/A

offline
ubuntu-test-VM-1

N/A

offline
D1-debian-FreshInstall-1

N/A

offline
D3-Debian-Slave-1

N/A

offline
ubuntu-test-VM-3

Channel reader thread: channel

"Channel reader thread: channel" Id=8 Group=main RUNNABLE (in native)
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:236)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    -  locked java.io.BufferedInputStream@12986ef
    at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
    at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

main

"main" Id=1 Group=main WAITING on hudson.remoting.Channel@18d107f
    at java.lang.Object.wait(Native Method)
    -  waiting on hudson.remoting.Channel@18d107f
    at java.lang.Object.wait(Object.java:502)
    at hudson.remoting.Channel.join(Channel.java:800)
    at hudson.remoting.Launcher.main(Launcher.java:484)
    at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430)
    at hudson.remoting.Launcher.run(Launcher.java:221)
    at hudson.remoting.Launcher.main(Launcher.java:180)

Ping thread for channel hudson.remoting.Channel@18d107f:channel

"Ping thread for channel hudson.remoting.Channel@18d107f:channel" Id=9 Group=main TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at hudson.remoting.PingThread.run(PingThread.java:86)

Pipe writer thread: channel

"Pipe writer thread: channel" Id=11 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

pool-1-thread-3

"pool-1-thread-3" Id=14 Group=main RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:392)
    at hudson.Functions.getThreadInfos(Functions.java:1061)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96)
    at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@5e305f

pool-1-thread-4

"pool-1-thread-4" Id=15 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:453)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:352)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

Finalizer

"Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@159cf26
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.ReferenceQueue$Lock@159cf26
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

Reference Handler

"Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@12efed9
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@12efed9
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

Signal Dispatcher

"Signal Dispatcher" Id=4 Group=system RUNNABLE
timja commented 11 years ago

elordahl:

I saw this for the first time today, and saw it twice. Three things have changed over the past week that could possibly affect this. I'm wondering how many people who have this problem use an NFS share, for I've seen a lot of file contention issues since the change to NFS.

1. (Few weeks ago)$JENKINS_HOME is now an NFS share
2. (Last night)Updated to 1.516
3. (Last night)Modified github plugin to use a, already-defined, ThreadFactory (see below):

-private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Executors.newCachedThreadPool());
+private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Computer.threadPoolForRemoting);

timja commented 11 years ago

jglick:

@walterk82 indeed -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true really only turns off the specialized error reporting, not the filter as a whole. For that you need to use -Dorg.kohsuke.stapler.jelly.DefaultScriptInvoker.compress=false from what I can tell.

timja commented 11 years ago

jglick:

Looking around for alternate implementations of GZIPOutputStream. JZlib looks most promising. Swapping this in for the native JRE implementation (zlib) should be trivial enough, but I need someone who is able to reproduce this bug and is willing to build Jenkins from sources (or give me a target Jenkins release version that I can build an alternate binary of) and compare the JZlib performance, so see if the hypothesis is valid that a bug in zlib is causing this issue.

timja commented 11 years ago

jglick:

Created http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.513.JENKINS-14362-jzlib/jenkins-war-1.513.JENKINS-14362-jzlib.war based on 1.513 if anyone wants to give it a whirl.

timja commented 11 years ago

marc_guenther:

guykisel: could you please wrap your stacktrace inside a {quote}, so this ticket remains readable?
EDIT: Thanks a lot

timja commented 11 years ago

guykisel:

Fixed. Apologies.

timja commented 11 years ago

jglick:

Thanks @guykisel; also in the future consider just attaching files.

timja commented 11 years ago

jglick:

Filed https://github.com/stapler/stapler/pull/22 and https://github.com/jenkinsci/jenkins/pull/804.

timja commented 11 years ago

jglick:

Upgrading priority since this has been reported to necessitate Jenkins restarts to avoid excess load after a few days.

timja commented 11 years ago

elordahl:

I've been using the "Monitoring" plugin to terminate the corresponding thread, instead of restarting Jenkins. The thread will be named something like "requestHandlerThread[#xyz]" and will most likely have the most cpu/user time by a long shot.

timja commented 11 years ago

elordahl:

@jglick - Can you build a 1.516 WAR with the fix? If so,I can test that Friday morning. I don't know what specifically is triggering the issue in our instance, but it repeatedly occurs during a specific segment of our build cycle.

timja commented 11 years ago

jglick:

@elordahl: http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.516.JENKINS-14362-jzlib/jenkins-war-1.516.JENKINS-14362-jzlib.war

timja commented 11 years ago

elordahl:

With the jenkins-war-1.516.JENKINS-14362-jzlib.war running, we did not see the hanging threads or CPUs @ 100%. Good sign. The next time I'll be able to test again will be Tuesday morning.

I'll also gladly install a 1.518 build w/ this fix as 1.518 has the css issue resolved @jglick - Let me know if you build one.

timja commented 11 years ago

joncrooke:

jenkins-war-1.516.JENKINS-14362-jzlib.war doesn't resolve this issue for me. CPU usage still peaks and Jenkins becomes locked out. Here's my stack trace:

2013-06-17 12:37:37
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.45-b01-451 mixed mode):

"Attach Listener" daemon prio=9 tid=7ff14f801000 nid=0x11c9b5000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"RequestHandlerThread[#12]" daemon prio=5 tid=7ff14e806000 nid=0x11f57b000 waiting on condition [11f57a000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"Handling GET /adjuncts/cb752a56/lib/layout/menu_right_arrow.png : RequestHandlerThread[#11]" daemon prio=5 tid=7ff14e9ed000 nid=0x11f926000 runnable [11f922000]
   java.lang.Thread.State: RUNNABLE
    at java.io.File.(File.java:318)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:996)
    at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:966)
    at sun.misc.URLClassPath.findResource(URLClassPath.java:146)
    at java.net.URLClassLoader$2.run(URLClassLoader.java:385)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findResource(URLClassLoader.java:382)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at jenkins.ClassLoaderReflectionToolkit.findResource(ClassLoaderReflectionToolkit.java:52)
    at hudson.PluginManager$UberClassLoader.findResource(PluginManager.java:972)
    at java.lang.ClassLoader.getResource(ClassLoader.java:1002)
    at org.kohsuke.stapler.framework.adjunct.AdjunctManager.allowResourceToBeServed(AdjunctManager.java:198)
    at org.kohsuke.stapler.framework.adjunct.AdjunctManager.doDynamic(AdjunctManager.java:165)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
    at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:363)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
    at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
    at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
    at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
    at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
    at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:680)

"RequestHandlerThread[#10]" daemon prio=5 tid=7ff14ff93000 nid=0x11c2f1000 waiting on condition [11c2f0000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"RequestHandlerThread[#9]" daemon prio=5 tid=7ff14ff92800 nid=0x11c8b2000 waiting on condition [11c8b1000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"pool-10-thread-1" prio=5 tid=7ff14ff37800 nid=0x11f823000 waiting on condition [11f822000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77e07af48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"JmDNS(Cornelius-Builder-Monkey-local.local.).State.Timer" prio=5 tid=7ff14cc94000 nid=0x12037d000 in Object.wait() [12037c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77e5be5c0> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <77e5be5c0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"JmDNS(Cornelius-Builder-Monkey-local.local.).Timer" daemon prio=5 tid=7ff14cada800 nid=0x11cabd000 runnable [11cabc000]
   java.lang.Thread.State: RUNNABLE
    at javax.jmdns.impl.DNSCache.allValues(DNSCache.java:349)
    - locked <77e5be668> (a javax.jmdns.impl.DNSCache)
    at javax.jmdns.impl.JmDNSImpl.cleanCache(JmDNSImpl.java:1771)
    at javax.jmdns.impl.tasks.RecordReaper.run(RecordReaper.java:58)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)

"pool-16-thread-1" daemon prio=5 tid=7ff14ce6e800 nid=0x120177000 waiting on condition [120176000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77e686eb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"Executor #1 for master" prio=5 tid=7ff14ce6d800 nid=0x120074000 in Object.wait() [120073000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77da4b940> (a hudson.model.Queue)
    at java.lang.Object.wait(Object.java:485)
    at hudson.util.OneShotEvent.block(OneShotEvent.java:72)
    - locked <77da4b940> (a hudson.model.Queue)
    at hudson.model.Queue.pop(Queue.java:882)
    - locked <77da4b940> (a hudson.model.Queue)
    at hudson.model.Executor.grabJob(Executor.java:289)
    at hudson.model.Executor.run(Executor.java:210)
    - locked <77da4b940> (a hudson.model.Queue)

"Jenkins UDP 33848 monitoring thread" prio=5 tid=7ff14ce69800 nid=0x11fd6b000 runnable [11fd6a000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainDatagramSocketImpl.receive0(Native Method)
    - locked <77e466810> (a java.net.PlainDatagramSocketImpl)
    at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
    - locked <77e466810> (a java.net.PlainDatagramSocketImpl)
    at java.net.DatagramSocket.receive(DatagramSocket.java:725)
    - locked <77e4ee470> (a java.net.DatagramPacket)
    - locked <77e6128e8> (a java.net.MulticastSocket)
    at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82)

"TCP slave agent listener port=0" prio=5 tid=7ff14ca42800 nid=0x11f478000 runnable [11f477000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    - locked <77e64d808> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91)

"NioSocketAcceptor-1" prio=5 tid=7ff14d0d8800 nid=0x11f2db000 runnable [11f2da000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
    at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
    at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <77e50f7d8> (a sun.nio.ch.Util$2)
    - locked <77e50f7c0> (a java.util.Collections$UnmodifiableSet)
    - locked <77e5f2d88> (a sun.nio.ch.KQueueSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238)
    at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432)
    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:680)

"RequestHandlerThread[#5]" daemon prio=5 tid=7ff14d028000 nid=0x11f720000 waiting on condition [11f71f000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
    at java.lang.Thread.run(Thread.java:680)

"jrobin " daemon prio=5 tid=7ff14d889800 nid=0x119f93000 in Object.wait() [119f92000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77dcfd2b0> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <77dcfd2b0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"DestroyJavaVM" prio=5 tid=7ff14bdb8800 nid=0x1103ce000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"LauncherControlThread[ControlPort=-1]" prio=5 tid=7ff14bdb7800 nid=0x11c6d3000 waiting on condition [11c6d2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at winstone.Launcher.run(Launcher.java:265)
    at java.lang.Thread.run(Thread.java:680)

"ConnectorThread:[http-8080]" daemon prio=5 tid=7ff14cea9000 nid=0x11c5d0000 runnable [11c5cf000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    - locked <77d8b0530> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at winstone.HttpListener.run(HttpListener.java:139)
    at java.lang.Thread.run(Thread.java:680)

"WinstoneHostConfigurationMgmt:default" daemon prio=5 tid=7ff14cf30000 nid=0x11c4cd000 waiting on condition [11c4cc000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at winstone.HostConfiguration.run(HostConfiguration.java:174)
    at java.lang.Thread.run(Thread.java:680)

"Java2D Disposer" daemon prio=10 tid=7ff14bd19800 nid=0x11ac46000 in Object.wait() [11ac45000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77db5c678> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <77db5c678> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at sun.java2d.Disposer.run(Disposer.java:127)
    at java.lang.Thread.run(Thread.java:680)

"AWT-AppKit" daemon prio=5 tid=7ff14ce3d000 nid=0x7fff74186180 runnable [00000000]
   java.lang.Thread.State: RUNNABLE

"Low Memory Detector" daemon prio=5 tid=7ff14c87f000 nid=0x119b64000 runnable [00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=9 tid=7ff14c87e000 nid=0x119a61000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=9 tid=7ff14c87d800 nid=0x11995e000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=9 tid=7ff14c87c800 nid=0x11985b000 runnable [00000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7ff14c87b800 nid=0x119758000 waiting on condition [00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=7ff14b840800 nid=0x1194ca000 in Object.wait() [1194c9000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77d875d58> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <77d875d58> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)

"Reference Handler" daemon prio=10 tid=7ff14b83f800 nid=0x1193c7000 in Object.wait() [1193c6000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <77d86b5a8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <77d86b5a8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=9 tid=7ff14b83d000 nid=0x1192c4000 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7ff14c801800 nid=0x1137d2000 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7ff14c802000 nid=0x1138d5000 runnable 

"Concurrent Mark-Sweep GC Thread" prio=9 tid=7ff14c84d000 nid=0x118f3e000 runnable 
"VM Periodic Task Thread" prio=10 tid=7ff14c888800 nid=0x119c67000 waiting on condition 

"Exception Catcher Thread" prio=10 tid=7ff14c801000 nid=0x1105f9000 runnable 
JNI global references: 1556
timja commented 11 years ago

jglick:

@joncrooke your issue looks unrelated and should be filed separately. Use your browser’s debug view to see if /adjuncts/cb752a56/lib/layout/menu_right_arrow.png is being served with an appropriate cache header; ought to be loaded once as a 200 and then get 304 NOT MODIFIED thereafter.

timja commented 11 years ago

jglick:

@elordahl: http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.518.JENKINS-14362-jzlib/jenkins-war-1.518.JENKINS-14362-jzlib.war

timja commented 11 years ago

dogfood:

Integrated in jenkins_main_trunk #2629
[FIXED JENKINS-17713 JENKINS-14362] (Revision bda95c67a6780ff0e6350f80d6f69cf0f61e737c)

Result = SUCCESS
kohsuke : bda95c67a6780ff0e6350f80d6f69cf0f61e737c
Files :

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/pom.xml
http://jenkins-ci.org/commit/jenkins/bda95c67a6780ff0e6350f80d6f69cf0f61e737c
Log:
[FIXED JENKINS-17713 JENKINS-14362]

Integrated the new version of Stapler that fixes them.

Compare: https://github.com/jenkinsci/jenkins/compare/59d4c6a4c25a...bda95c67a678

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
cli/pom.xml
core/pom.xml
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/console/AnnotatedLargeText.java
core/src/main/java/hudson/console/ConsoleNote.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/UsageStatistics.java
core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
core/src/main/java/hudson/util/CompressedFile.java
maven-plugin/pom.xml
plugins/pom.xml
pom.xml
test/pom.xml
ui-samples-plugin/pom.xml
war/pom.xml
http://jenkins-ci.org/commit/jenkins/3701a327a75d35e5d0e5795341cc55bb123a2007
Log:
JENKINS-14362 Switch to jzlib for GZIP streaming.

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
cli/pom.xml
cli/src/main/java/hudson/cli/CLI.java
core/pom.xml
core/src/main/java/hudson/AboutJenkins.java
core/src/main/java/hudson/ClassicPluginStrategy.java
core/src/main/java/hudson/ExtensionFinder.java
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/Functions.java
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/main/java/hudson/WebAppMain.java
core/src/main/java/hudson/cli/ClientAuthenticationCache.java
core/src/main/java/hudson/cli/CommandDuringBuild.java
core/src/main/java/hudson/cli/SetBuildParameterCommand.java
core/src/main/java/hudson/init/impl/GroovyInitScript.java
core/src/main/java/hudson/logging/LogRecorder.java
core/src/main/java/hudson/markup/MyspacePolicy.java
core/src/main/java/hudson/matrix/Combination.java
core/src/main/java/hudson/matrix/DefaultMatrixExecutionStrategyImpl.java
core/src/main/java/hudson/matrix/MatrixBuild.java
core/src/main/java/hudson/matrix/MatrixConfiguration.java
core/src/main/java/hudson/matrix/MatrixProject.java
core/src/main/java/hudson/model/AbstractBuild.java
core/src/main/java/hudson/model/AbstractItem.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/Cause.java
core/src/main/java/hudson/model/CauseAction.java
core/src/main/java/hudson/model/Computer.java
core/src/main/java/hudson/model/ComputerSet.java
core/src/main/java/hudson/model/Descriptor.java
core/src/main/java/hudson/model/DisplayNameListener.java
core/src/main/java/hudson/model/Executor.java
core/src/main/java/hudson/model/FileParameterDefinition.java
core/src/main/java/hudson/model/FileParameterValue.java
core/src/main/java/hudson/model/Fingerprint.java
core/src/main/java/hudson/model/ItemGroup.java
core/src/main/java/hudson/model/JDK.java
core/src/main/java/hudson/model/Job.java
core/src/main/java/hudson/model/ListView.java
core/src/main/java/hudson/model/Node.java
core/src/main/java/hudson/model/ParametersAction.java
core/src/main/java/hudson/model/ParametersDefinitionProperty.java
core/src/main/java/hudson/model/Queue.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/RunAction.java
core/src/main/java/hudson/model/RunMap.java
core/src/main/java/hudson/model/RunParameterDefinition.java
core/src/main/java/hudson/model/RunParameterValue.java
core/src/main/java/hudson/model/StreamBuildListener.java
core/src/main/java/hudson/model/UpdateCenter.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/queue/CauseOfBlockage.java
core/src/main/java/hudson/model/queue/WorkUnit.java
core/src/main/java/hudson/os/PosixAPI.java
core/src/main/java/hudson/os/PosixException.java
core/src/main/java/hudson/scheduler/CronTab.java
core/src/main/java/hudson/search/Search.java
core/src/main/java/hudson/search/SearchItem.java
core/src/main/java/hudson/security/BasicAuthenticationFilter.java
core/src/main/java/hudson/security/FederatedLoginService.java
core/src/main/java/hudson/security/HudsonPrivateSecurityRealm.java
core/src/main/java/hudson/security/Permission.java
core/src/main/java/hudson/slaves/SlaveComputer.java
core/src/main/java/hudson/tasks/Fingerprinter.java
core/src/main/java/hudson/tasks/Maven.java
core/src/main/java/hudson/tasks/UserAvatarResolver.java
core/src/main/java/hudson/tasks/junit/CaseResult.java
core/src/main/java/hudson/tasks/junit/ClassResult.java
core/src/main/java/hudson/tasks/junit/JUnitParser.java
core/src/main/java/hudson/tasks/junit/PackageResult.java
core/src/main/java/hudson/tasks/junit/TestNameTransformer.java
core/src/main/java/hudson/tasks/test/TestResult.java
core/src/main/java/hudson/tools/CommandInstaller.java
core/src/main/java/hudson/tools/ZipExtractionInstaller.java
core/src/main/java/hudson/util/CopyOnWriteMap.java
core/src/main/java/hudson/util/FormFieldValidator.java
core/src/main/java/hudson/util/FormValidation.java
core/src/main/java/hudson/util/IOUtils.java
core/src/main/java/hudson/util/PersistedList.java
core/src/main/java/hudson/util/RunList.java
core/src/main/java/hudson/util/XStream2.java
core/src/main/java/hudson/util/jna/GNUCLibrary.java
core/src/main/java/hudson/widgets/Widget.java
core/src/main/java/jenkins/model/Jenkins.java
core/src/main/java/jenkins/model/PeepholePermalink.java
core/src/main/java/jenkins/model/RunAction2.java
core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java
core/src/main/java/jenkins/model/lazy/Boundary.java
core/src/main/java/jenkins/model/lazy/BuildReferenceMapAdapter.java
core/src/main/java/jenkins/mvn/GlobalMavenConfig.java
core/src/main/java/jenkins/security/ConfidentialStore.java
core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java
core/src/main/java/jenkins/util/JSONSignatureValidator.java
core/src/main/java/jenkins/util/ProgressiveRendering.java
core/src/main/java/jenkins/widgets/BuildQueueWidget.java
core/src/main/java/jenkins/widgets/ExecutorsWidget.java
core/src/main/resources/dsld/IntelliJ.gdsl
core/src/main/resources/hudson/AboutJenkins/index.jelly
core/src/main/resources/hudson/AboutJenkins/index.properties
core/src/main/resources/hudson/AboutJenkins/index_fr.properties
core/src/main/resources/hudson/Messages.properties
core/src/main/resources/hudson/Messages_de.properties
core/src/main/resources/hudson/Messages_fr.properties
core/src/main/resources/hudson/PluginManager/advanced.jelly
core/src/main/resources/hudson/PluginManager/installed.jelly
core/src/main/resources/hudson/cli/Messages.properties
core/src/main/resources/hudson/logging/LogRecorder/configure.jelly
core/src/main/resources/hudson/logging/LogRecorder/index.jelly
core/src/main/resources/hudson/matrix/MatrixBuild/ajaxMatrix_fr.properties
core/src/main/resources/hudson/matrix/MatrixProject/ajaxMatrix_fr.properties
core/src/main/resources/hudson/matrix/MatrixProject/index.jelly
core/src/main/resources/hudson/matrix/Messages.properties
core/src/main/resources/hudson/model/Cause/UpstreamCause/description_pl.properties
core/src/main/resources/hudson/model/Cause/UserCause/description_pl.properties
core/src/main/resources/hudson/model/Cause/UserIdCause/description_pl.properties
core/src/main/resources/hudson/model/FileParameterDefinition/config_fr.properties
core/src/main/resources/hudson/model/FileParameterValue/value_fr.properties
core/src/main/resources/hudson/model/FreeStyleProject/newJobDetail_fr.properties
core/src/main/resources/hudson/model/Job/_api.jelly
core/src/main/resources/hudson/model/Job/buildTimeTrend_fr.properties
core/src/main/resources/hudson/model/Job/configure_fr.properties
core/src/main/resources/hudson/model/Job/index.jelly
core/src/main/resources/hudson/model/Job/index_fr.properties
core/src/main/resources/hudson/model/Messages.properties
core/src/main/resources/hudson/model/Messages_de.properties
core/src/main/resources/hudson/model/Messages_fr.properties
core/src/main/resources/hudson/model/ParametersAction/index_fr.properties
core/src/main/resources/hudson/model/ParametersDefinitionProperty/index_fr.properties
core/src/main/resources/hudson/model/PasswordParameterDefinition/config_fr.properties
core/src/main/resources/hudson/model/RunParameterDefinition/config.jelly
core/src/main/resources/hudson/model/RunParameterDefinition/index.jelly
core/src/main/resources/hudson/model/View/AsynchPeople/index.jelly
core/src/main/resources/hudson/model/View/AsynchPeople/index.properties
core/src/main/resources/hudson/model/View/sidepanel.jelly
core/src/main/resources/hudson/model/labels/LabelAtom/configure_fr.properties
core/src/main/resources/hudson/slaves/JNLPLauncher/main.jelly
core/src/main/resources/hudson/slaves/Messages_fr.properties
core/src/main/resources/hudson/slaves/OfflineCause/LaunchFailed/cause_fr.properties
core/src/main/resources/hudson/slaves/SlaveComputer/log.jelly
core/src/main/resources/hudson/slaves/SlaveComputer/statistics.groovy
core/src/main/resources/hudson/tasks/BuildTrigger/config.jelly
core/src/main/resources/hudson/tasks/Fingerprinter/FingerprintAction/index.jelly
core/src/main/resources/hudson/tasks/Maven/config.jelly
core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly
core/src/main/resources/hudson/tasks/junit/ClassResult/body.jelly
core/src/main/resources/hudson/tasks/test/AbstractTestResultAction/summary.jelly
core/src/main/resources/hudson/tasks/test/MetaTabulatedResult/body.jelly
core/src/main/resources/hudson/tools/JDKInstaller/config.jelly
core/src/main/resources/hudson/tools/label.jelly
core/src/main/resources/hudson/views/JobColumn/column.jelly
core/src/main/resources/jenkins/diagnostics/SecurityIsOffMonitor/message_fr.properties
core/src/main/resources/jenkins/management/Messages_pl.properties
core/src/main/resources/jenkins/management/PluginsLink/info_pl.properties
core/src/main/resources/jenkins/model/CauseOfInterruption/UserInterruption/summary_pl.properties
core/src/main/resources/jenkins/model/Jenkins/legend_fr.properties
core/src/main/resources/jenkins/model/Jenkins/manage.jelly
core/src/main/resources/jenkins/mvn/GlobalMavenConfig/config.groovy
core/src/main/resources/jenkins/widgets/BuildQueueWidget/index.groovy
core/src/main/resources/jenkins/widgets/ExecutorsWidget/index.groovy
core/src/main/resources/lib/form/apply_fr.properties
core/src/main/resources/lib/form/booleanRadio_fr.properties
core/src/main/resources/lib/form/breadcrumb-config-outline_fr.properties
core/src/main/resources/lib/form/dropdownDescriptorSelector.jelly
core/src/main/resources/lib/form/hetero-list_fr.properties
core/src/main/resources/lib/form/number.jelly
core/src/main/resources/lib/form/textarea.jelly
core/src/main/resources/lib/hudson/buildListTable.jelly
core/src/main/resources/lib/hudson/executors.jelly
core/src/main/resources/lib/hudson/jobLink.jelly
core/src/main/resources/lib/hudson/queue.jelly
core/src/main/resources/lib/layout/breadcrumbBar_fr.properties
core/src/main/resources/lib/layout/breadcrumbs.js
core/src/main/resources/lib/layout/breakable.jelly
core/src/main/resources/lib/layout/copyButton/copyButton.js
core/src/main/resources/lib/layout/layout.jelly
core/src/main/resources/lib/layout/layout_fr.properties
core/src/main/resources/lib/layout/progressiveRendering.jelly
core/src/main/resources/lib/layout/progressiveRendering/progressiveRendering.js
core/src/main/resources/lib/layout/progressiveRendering_fr.properties
core/src/main/resources/lib/layout/task_fr.properties
core/src/main/resources/windows-service/jenkins-slave.xml
core/src/main/resources/windows-service/jenkins.xml
core/src/test/java/hudson/FilePathTest.java
core/src/test/java/hudson/FunctionsTest.java
core/src/test/java/hudson/logging/LogRecorderTest.java
core/src/test/java/hudson/markup/MyspacePolicyTest.java
core/src/test/java/hudson/matrix/CombinationFilterUsingBuildParamsTest.java
core/src/test/java/hudson/model/FingerprintTest.java
core/src/test/java/hudson/model/ListViewTest.java
core/src/test/java/hudson/model/StubJob.java
core/src/test/java/hudson/tasks/junit/SuiteResultTest.java
core/src/test/java/hudson/util/CopyOnWriteMapTest.java
core/src/test/java/jenkins/model/PeepholePermalinkTest.java
core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java
core/src/test/resources/hudson/model/fingerprint.xml
debian/debian/changelog
maven-plugin/pom.xml
maven-plugin/src/main/java/hudson/maven/AbstractMavenProcessFactory.java
maven-plugin/src/main/java/hudson/maven/AggregatingClassLoader.java
maven-plugin/src/main/java/hudson/maven/Maven3Builder.java
maven-plugin/src/main/java/hudson/maven/Maven3ProcessFactory.java
maven-plugin/src/main/java/hudson/maven/MavenBuildInformation.java
maven-plugin/src/main/java/hudson/maven/MavenEmbedderRequest.java
maven-plugin/src/main/java/hudson/maven/MavenModuleSet.java
maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
maven-plugin/src/main/java/hudson/maven/MavenProbeAction.java
maven-plugin/src/main/java/hudson/maven/MavenProcessFactory.java
maven-plugin/src/main/java/hudson/maven/MavenUtil.java
maven-plugin/src/main/java/hudson/maven/PlexusModuleContributor.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenFingerprinter.java
maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java
maven-plugin/src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
maven-plugin/src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
maven-plugin/src/main/resources/hudson/maven/Messages.properties
maven-plugin/src/main/resources/hudson/maven/Messages_es.properties
maven-plugin/src/main/resources/hudson/maven/Messages_zh_TW.properties
maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
maven-plugin/src/test/java/hudson/maven/reporters/TestMojoTest.java
plugins/pom.xml
pom.xml
test/pom.xml
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
test/src/main/java/org/jvnet/hudson/test/MemoryAssert.java
test/src/main/java/org/jvnet/hudson/test/MockFolder.java
test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java
test/src/main/resources/org/jvnet/hudson/test/ComputerConnectorTester/configure_fr.properties
test/src/test/groovy/hudson/cli/BuildCommandTest.groovy
test/src/test/groovy/hudson/cli/SetBuildParameterCommandTest.groovy
test/src/test/java/hudson/PluginManagerTest.java
test/src/test/java/hudson/cli/CopyJobCommandTest.java
test/src/test/java/hudson/init/impl/GroovyInitScriptTest.java
test/src/test/java/hudson/logging/LogRecorderManagerTest.java
test/src/test/java/hudson/matrix/MatrixTest.java
test/src/test/java/hudson/maven/MavenModuleSetTest.java
test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java
test/src/test/java/hudson/maven/MavenMultiModuleTest.java
test/src/test/java/hudson/maven/MavenProjectTest.java
test/src/test/java/hudson/maven/PlexusModuleContributorTest.java
test/src/test/java/hudson/model/AbstractProjectTest.java
test/src/test/java/hudson/model/DisplayNameTest.java
test/src/test/java/hudson/model/JobTest.java
test/src/test/java/hudson/model/ListViewTest.java
test/src/test/java/hudson/model/ParametersTest.java
test/src/test/java/hudson/model/RunParameterDefinitionTest.java
test/src/test/java/hudson/model/RunTest.java
test/src/test/java/hudson/tasks/FingerprinterTest.java
test/src/test/java/hudson/tasks/MavenTest.java
test/src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
test/src/test/java/hudson/tasks/junit/SuiteResultTest.java
test/src/test/java/hudson/tasks/junit/TestNameTransformerTest.java
test/src/test/java/hudson/tasks/junit/TestResultPublishingTest.java
test/src/test/java/jenkins/model/JenkinsTest.java
test/src/test/java/jenkins/security/RekeySecretAdminMonitorTest.java
test/src/test/java/org/jvnet/hudson/main/UseRecipesWithJenkinsRuleTest.java
test/src/test/resources/hudson/init/impl/GroovyInitScriptTest/errorsHandled.zip
test/src/test/resources/hudson/maven/custom-plexus-component.pom
test/src/test/resources/hudson/model/JobTest/configDotXmlPermission.zip
test/src/test/resources/hudson/model/JobTest/getArtifactsUpTo.zip
test/src/test/resources/hudson/model/JobTest/readPermission.zip
test/src/test/resources/hudson/model/JobTest/testConfigDotXmlPermission.zip
test/src/test/resources/hudson/model/JobTest/testGetArtifactsUpTo.zip
test/src/test/resources/hudson/model/JobTest/testReadPermission.zip
test/src/test/resources/hudson/model/ListViewTest/nullJobNames/config.xml
test/src/test/resources/hudson/tasks/FingerprinterTest/actionSerialization.zip
ui-samples-plugin/pom.xml
ui-samples-plugin/src/main/resources/index.jelly
war/pom.xml
war/src/main/webapp/css/style.css
war/src/main/webapp/help/parameter/file.html
war/src/main/webapp/help/parameter/run-filter.html
war/src/main/webapp/help/parameter/run-project.html
war/src/main/webapp/help/project-config/description.html
war/src/main/webapp/help/run-config/description.html
war/src/main/webapp/help/system-config/systemMessage.html
war/src/main/webapp/help/user/description.html
war/src/main/webapp/help/view-config/description.html
war/src/main/webapp/scripts/hudson-behavior.js
http://jenkins-ci.org/commit/jenkins/dedb24ef034a04fea288eaa50089e8e774714e99
Log:
Merge branch 'master' into JENKINS-14362-jzlib

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/pom.xml
core/src/main/java/hudson/ExtensionListView.java
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/Functions.java
core/src/main/java/hudson/Launcher.java
core/src/main/java/hudson/TcpSlaveAgentListener.java
core/src/main/java/hudson/Util.java
core/src/main/java/hudson/cli/CliProtocol.java
core/src/main/java/hudson/cli/ListJobsCommand.java
core/src/main/java/hudson/model/AbstractBuild.java
core/src/main/java/hudson/model/AbstractProject.java
core/src/main/java/hudson/model/Computer.java
core/src/main/java/hudson/model/ComputerSet.java
core/src/main/java/hudson/model/Executor.java
core/src/main/java/hudson/model/Fingerprint.java
core/src/main/java/hudson/model/Label.java
core/src/main/java/hudson/model/Node.java
core/src/main/java/hudson/model/ParametersDefinitionProperty.java
core/src/main/java/hudson/model/Queue.java
core/src/main/java/hudson/model/RestartListener.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/Slave.java
core/src/main/java/hudson/model/View.java
core/src/main/java/hudson/model/listeners/RunListener.java
core/src/main/java/hudson/model/queue/AbstractQueueTask.java
core/src/main/java/hudson/model/queue/CauseOfBlockage.java
core/src/main/java/hudson/model/queue/MappingWorksheet.java
core/src/main/java/hudson/model/queue/QueueListener.java
core/src/main/java/hudson/model/queue/SubTask.java
core/src/main/java/hudson/model/queue/SubTaskContributor.java
core/src/main/java/hudson/model/queue/Tasks.java
core/src/main/java/hudson/node_monitors/AbstractDiskSpaceMonitor.java
core/src/main/java/hudson/node_monitors/AbstractNodeMonitorDescriptor.java
core/src/main/java/hudson/node_monitors/DiskSpaceMonitor.java
core/src/main/java/hudson/node_monitors/DiskSpaceMonitorDescriptor.java
core/src/main/java/hudson/node_monitors/NodeMonitor.java
core/src/main/java/hudson/node_monitors/TemporarySpaceMonitor.java
core/src/main/java/hudson/security/GlobalSecurityConfiguration.java
core/src/main/java/hudson/slaves/CloudProvisioningListener.java
core/src/main/java/hudson/slaves/NodePropertyDescriptor.java
core/src/main/java/hudson/slaves/NodeProvisioner.java
core/src/main/java/hudson/slaves/SlaveComputer.java
core/src/main/java/hudson/tasks/junit/CaseResult.java
core/src/main/java/hudson/tasks/junit/ClassResult.java
core/src/main/java/hudson/tasks/junit/PackageResult.java
core/src/main/java/hudson/util/CopyOnWriteList.java
core/src/main/java/hudson/util/PersistedList.java
core/src/main/java/jenkins/model/Jenkins.java
core/src/main/java/jenkins/security/QueueItemAuthenticator.java
core/src/main/java/jenkins/security/QueueItemAuthenticatorConfiguration.java
core/src/main/java/jenkins/security/QueueItemAuthenticatorDescriptor.java
core/src/main/java/jenkins/slaves/JnlpSlaveAgentProtocol.java
core/src/main/java/jenkins/util/TimeDuration.java
core/src/main/resources/hudson/matrix/Messages_de.properties
core/src/main/resources/hudson/model/AbstractProject/sidepanel.jelly
core/src/main/resources/hudson/model/Messages.properties
core/src/main/resources/hudson/model/Messages_de.properties
core/src/main/resources/hudson/security/Messages_de.properties
core/src/main/resources/hudson/views/BuildButtonColumn/column.jelly
core/src/main/resources/jenkins/model/GlobalNodePropertiesConfiguration/config.groovy
core/src/main/resources/jenkins/security/QueueItemAuthenticator/config.groovy
core/src/main/resources/jenkins/security/QueueItemAuthenticatorConfiguration/config.groovy
core/src/main/resources/lib/hudson/projectView.jelly
core/src/main/resources/lib/layout/confirmationLink.jelly
core/src/main/resources/lib/layout/task.jelly
core/src/test/java/hudson/LauncherTest.java
core/src/test/java/hudson/cli/ListJobsCommandTest.java
core/src/test/java/hudson/model/ViewTest.java
core/src/test/java/hudson/tasks/junit/ClassResultTest.java
debian/debian/changelog
maven-plugin/src/main/resources/hudson/maven/Messages.properties
maven-plugin/src/main/resources/hudson/maven/Messages_de.properties
maven-plugin/src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
pom.xml
test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java
test/src/test/java/hudson/maven/MavenMultiModuleTest.java
test/src/test/java/hudson/model/QueueTest.java
test/src/test/java/hudson/model/ViewTest.java
http://jenkins-ci.org/commit/jenkins/b7e2d6127f77711681113a020681508d57622ea5
Log:
Merge branch 'master' into JENKINS-14362-jzlib

Conflicts:
core/pom.xml

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/console/AnnotatedLargeText.java
core/src/main/java/hudson/console/ConsoleNote.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/model/UsageStatistics.java
core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java
core/src/main/java/hudson/util/CompressedFile.java
http://jenkins-ci.org/commit/jenkins/59ff3514532f9c7faef2bc6d841aba53a8107f0f
Log:
Merge pull request #804 from jglick/JENKINS-14362-jzlib

Use jzlib

Compare: https://github.com/jenkinsci/jenkins/compare/c1c6475bbb1a...59ff3514532f

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332
Log:
JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk.

timja commented 11 years ago

jglick:

Fix expected in 1.520.

timja commented 11 years ago

dogfood:

Integrated in jenkins_main_trunk #2633
JENKINS-14362 Switch to jzlib for GZIP streaming. (Revision 3701a327a75d35e5d0e5795341cc55bb123a2007)
JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk. (Revision fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332)

Result = SUCCESS
Jesse Glick : 3701a327a75d35e5d0e5795341cc55bb123a2007
Files :

Jesse Glick : fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332
Files :

timja commented 11 years ago

jglick:

http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.2.JENKINS-14362-jzlib/jenkins-war-1.509.2.JENKINS-14362-jzlib.war now also available if anyone using LTS thinks they are running into this.

timja commented 11 years ago

justinharringa:

Is there any chance that this might get backported besides just providing the WAR? I noticed that you already marked it as an lts-candidate but it didn't make it into 1.509.2 and wasn't rejected either. Our infra guys prefer installing via RPM.

timja commented 11 years ago

jglick:

@justinharringa: not much made it into 1.509.2, but yes this is a candidate for 1.509.3. The most helpful thing you can do to push it forward is to verify that the fix actually works. Since I have never personally observed the original bug I cannot do that, and we are loath to backport unverified fixes, though I have gotten anecdotal reports that it works.

Verifying the fix would mean confirming in a comment in this issue that (a) you consistently observed this problem (meaning heavy CPU load with thread dumps showing one or more threads stuck in Deflater.deflateBytes) in an earlier build, such as 1.519 or 1.509.2, but (b) you have not seen it recur since updating to a build with the purported fix (1.520 or 1.509.2.JENKINS-14362-jzlib).

timja commented 11 years ago

justinharringa:

That sounds good. I'll see what I can do. I thought I had reproduced the issue by opening the Wall Display plugin report in a few tabs. After that, I hit refresh rapidly on one of the tabs. Then, the CPU was pegged. Unfortunately, the CPU usage calmed down after a while and I didn't see a thread dump indicating the usage of Deflater.deflateBytes. This seemed to cause issues in one of our more heavily used instances but it appears it corrects itself in the sandbox environment so it could be a red herring. This doesn't appear to be an effective way to reproduce the issue.

I'm currently on 1.480.3. Once I find a way to reproduce, I'll try to reproduce on 1.509.2 as well and then move to your patched WAR in this sandbox environment.

timja commented 11 years ago

jglick:

the CPU was pegged

Remember that this by itself does not necessarily, or even probably, indicate that you are seeing JENKINS-14362; there are numerous other bugs that could cause inappropriate heavy CPU usage. You have to check the master thread dump for threads using Deflater.deflateBytes. In reports I have heard, the typical symptom is that Jenkins starts off running fine, but over the course of a week or so more and more threads get “stuck” in this method (seemingly at random), each consuming 100% of one CPU, so that eventually there are no free processors left to do anything.