jonahadmin
commented
8 years ago I think my interpretation of the problem may be incorrect. The issue is inconsistent. I'll follow up if I have better information.
Closed jonahadmin closed 8 years ago
jonahadmin
commented
8 years ago I think my interpretation of the problem may be incorrect. The issue is inconsistent. I'll follow up if I have better information.
jonahadmin
commented
8 years ago I am unable to determine any kind of pattern at all. I've tried re-arranging build steps, inserting random build steps in case one step was affecting others, but the problem continues to happen completely randomly. It is happening with both the SSH Exec and the SSHDeployer plugins, and it's always the same error, with no other information.
[2016-06-24 17:07:38,597] ERROR - jetbrains.buildServer.AGENT - com.jcraft.jsch.JSchException: session is down
jetbrains.buildServer.RunBuildException: com.jcraft.jsch.JSchException: session is down
at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.runProcess(ScpProcessAdapter.java:93)
at jetbrains.buildServer.deployer.agent.SyncBuildProcessAdapter.start(SyncBuildProcessAdapter.java:58)
at jetbrains.buildServer.agent.impl.buildStages.runnerStages.start.CallRunnerStage.doBuildStage(CallRunnerStage.java:58)
at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:25)
at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:18)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor.doStages(RunnerStagesExecutor.java:18)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunnerContextExecutor.callRunnerStages(RunnerContextExecutor.java:43)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.StepExecutor.processNextStep(StepExecutor.java:25)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeRunnerStep(ForEachBuildRunnerStage.java:138)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.runStep(ForEachBuildRunnerStage.java:123)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeBuildRunners(ForEachBuildRunnerStage.java:83)
at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.doBuildStage(ForEachBuildRunnerStage.java:44)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:288)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:53)
at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:252)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: session is down
at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:762)
at com.jcraft.jsch.Channel.connect(Channel.java:151)
at com.jcraft.jsch.Channel.connect(Channel.java:145)
at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.upload(ScpProcessAdapter.java:121)
at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.runProcess(ScpProcessAdapter.java:88)
... 24 moreSorry for the stream of consciousness style reporting, but we may have a new theory that there is something flakey going on with the server itself rather than the plugin. I tried performing similar commands using bonafied ssh, and I was getting similarly strange results including write failures. Apparently RHEL7 prior to 7.2 did have ssh bugs. We've just done a full update and I guess we'll see what happens.
jonahadmin
commented
8 years ago FYI, for anyone reading this thread, the server in question is an Amazon Redhat 7.2 AMI. We just did a yum update, and now we'll have to wait to see if it solved the problem.
nskvortsov
commented
8 years ago Thank you for detailed follow up. Please, let us know if upgrading the server helps 25 июня 2016 г. 1:28 пользователь "jonahadmin" notifications@github.com написал:
FYI, for anyone reading this thread, the server in question is an Amazon Redhat 7.2 AMI. We just did a yum update, and now we'll have to wait to see if it solved the problem.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JetBrains/teamcity-deployer-plugin/issues/88#issuecomment-228478060, or mute the thread https://github.com/notifications/unsubscribe/AAg5kyPITAgQGrntx5c-IGJ7YXBR6buNks5qPFoJgaJpZM4I84qa .
nskvortsov
commented
8 years ago @jonahadmin so, how did upgrade go? Is it any better now?
jonahadmin
commented
8 years ago Since we did the yum update, the problem appears to have gone away complete. So it appears that the problem all along was that the base RHEL 7.2 AMI has some unexpected yet critical bugs with SSH.
I am in awe that they could create an AMI with such serious problems.
There seems to be an issue with com.jcraft.jsch.JSchException, or at least how it's being used.
I have two build separate build steps, that both connect to the same server. The first step executes find, but then any subsequent SSH Exec step results in the exception below.
Based on cursory research, I'm guessing that instead of trying to establish a new connection, it tries to somehow reuse the existing one even though it's already been closed.