[JENKINS-45290] windows slaves stops being able to connect to master

I need stactraces from both Jenkins Master and the agent for the time when the connection hangs.

timja commented 7 years ago

I have attached the log files.

timja commented 7 years ago

E.g. see https://social.msdn.microsoft.com/Forums/en-US/212feb8f-cf96-4561-9953-40a21d21ea47/exception-an-established-connection-was-aborted-by-the-software-in-your-host-machine?forum=netfxnetcom

Have you or your IT applied any security configuration changes on the machine? "An established connection was aborted by the software in your host machine" from antivirus or Firewall. So this is a system issue.

timja commented 7 years ago

That is indeed interesting, but doesn't seem to overlap with the problems we see.

After the issue occur, none of the windows slaves can connect from being in a disconnected state. We have 50+ windows slaves on this jenkins master.

If we restart the jenkins master, the problem goes away on all slaves. It seems very improbable to me that this would be a system issue based on that information.

We can also direct the slaves to a different master and they have no trouble connecting to that one.

I pasted the TCPdump in the description of this issue, showing all traffic sent over 5 minutes. The master received 4 packets, but sent none.

timja commented 7 years ago

In such case the connection termination probably makes the connection receiver to go into a bogus state. Curious thing is that it happens for both JNLP2 and JNLP4 which are supposed to use different NIO hubs. Would it be possible to get logs before 3:06 PM, probably there is a root cause error there. I would also appreciate a Master's thread dump for the time the agents cannot connect.

I also see another minor issue in REST API, will create a separate defect for it:

java.lang.Thread.run(Thread.java:748) Caused by: org.kohsuke.stapler.export.NotExportableException: class java.nio.channels.ClosedChannelException doesn't have @ExportedBean so cannot write hudson.slaves.OfflineCause$ChannelTermination.cause at org.kohsuke.stapler.export.Model.(Model.java:79) at org.kohsuke.stapler.export.ModelBuilder.get(ModelBuilder.java:51) at

timja commented 7 years ago

We have logs going back to January. I haven't tried, but I expect that the current log (from Jul 2, 2017 06.35 AM) is too large to just attach to this ticket. Uncompressed it is 24GB, compressed it should be between 2-3 GB.

How should I send this log file to you?

Can you please explain, or point me to documentation, on how to obtain the Master's thread dump?

timja commented 7 years ago

We have been forced to restart jenkins. As I mentioned earlier, this works as a temporary workaround since all windows slaves are able to connect again.

We will try to set up monitoring for the next time this issue occur so we have less logs to dig through. Are there any logs from a troubleshooting perspective that would be interesting to follow where we can't look up the information afterwards?

timja commented 7 years ago

> Can you please explain, or point me to documentation, on how to obtain the Master's thread dump?

You can use low-level tools like jstack, but I would rather recommend generating a support bundle using https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin

timja commented 7 years ago

I will install the support core plugin on Monday (July 10th).

timja commented 7 years ago

Support Core Plugin is now installed

timja commented 7 years ago

Waiting for the bundles then, thanks!

timja commented 7 years ago

ping

timja commented 7 years ago

Apologies for the late reply, I have been on vacation.

We are still waiting on the issue to reappear so we can generate the support bundles for you.

timja commented 7 years ago

Any update on this one? we are facing same issue here with Win Slaves.

timja commented 7 years ago

I cannot investigate the issue without master/agent threaddumps

ronen_tef if you are ready to provide them, it would help

timja commented 7 years ago

I have not experienced the issue since I installed the support core plugin.

I'm waiting for the issue too occur again so I can provide the logs necessary.

timja commented 7 years ago

oleg_nenashev could you please let me know how to take those threaddumps?

timja commented 7 years ago

Okay, I'm attaching 2 outputs of the Master thread dump,

one is via Jenkins console and one using java jconsole directly from the unix command line.

As for the Win slave I can't take any as the proccess even not started, the only thing I could provide is an exeption output.

timja commented 7 years ago

Any progress/estimation for a fix?

timja commented 7 years ago

ronen_tef an agent log would be useful just to have a full snapshots. If this is "java.io.IOException: An established connection was aborted by the software in your host machine" again, likely it's a whatever issue in your system.

I cannot provide any ETA until the bug is triaged at least. But I would not expect the fix in October even if the issue is confirmed (unless it's a quick fix).

timja commented 7 years ago

oleg_nenashev as already commented, I can't provide any agent log as the agent is not even started therfore a log is not created.

"...again, likely it's a whatever issue in your system...." in this case, how could it be that the same is not occur with ver. 2.46.1 ? just once upgrading to ver. 2.78.

The behavier should be consistent I assume.

timja commented 7 years ago

ronen_tef Currently I cannot confirm you have the same issue as ekerhag. If you are sure it's the same, you should be able to provide agent connection stdout/stderr at least (from either slave.jar STDOUT or wrapper logs).

timja commented 6 years ago