[JENKINS-26439] Build does not abort when «Send files or execute commands over SSH» fails

[jira-importer]

In case of an error, i.e. when files cannot be copied, it makes no sense to continue the build. But Jenkins does neither abort the build, when something failes, nor can I see any configuration for that behaviour.

e.g.:

SSH: Connecting from host [dev0004]
SSH: Connecting with configuration [web9t.int.swisssign.net] ...
SSH: EXEC: STDOUT/STDERR from command [ls /var/tmp/shop-certificate.*] ...
ls: cannot access /var/tmp/shop-certificate.*: No such file or directory
SSH: EXEC: completed after 200 ms
SSH: Disconnecting configuration [web9t.int.swisssign.net] ...
ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [2]]
Build step 'Send files or execute commands over SSH' changed build result to UNSTABLE
[workspace] $ /bin/bash -xe /tmp/hudson4848040255435626821.sh
+ cd webtester
+ unzip ../joomla-plugin-virtuemart-3.0.2.zip
Archive: ../joomla-plugin-virtuemart-3.0.2.zip
inflating: com_virtuemart.3.0.2.tar.gz
inflating: com_virtuemart.3.0.2_ext_aio.tar.gz
[workspace] $ /bin/bash -xe /tmp/hudson2548916983990174771.sh
+ cd webtester
+ (( i=0 ))
+ (( i<10 ))
+ xvfb-run a '-server-args=-screen 0 2048x2048x24' webrunner ping-joomla.wt
[... build continues ...]

---

Originally reported by mrw, imported from: Build does not abort when «Send files or execute commands over SSH» fails

• status: Open
• priority: Blocker
• resolution: Unresolved
• imported: 2022/01/10

[jira-importer]

mrw:

Configuration is following:

Source files: /var/lib/jenkins/keys/shop-certificate.*
Remove prefix:
Remote directory: /var/tmp/
Exec command: ls /var/tmp/shop-certificate.*

On the jenkins server:

jenkins@dev0004:~$ ls /var/lib/jenkins/keys/shop-certificate.*
/var/lib/jenkins/keys/shop-certificate.crt /var/lib/jenkins/keys/shop-certificate.pem
/var/lib/jenkins/keys/shop-certificate.key.pem

So problem #1 is, that files are not transfered - without any error message not abort.
That's why I added "Exec command: ls /var/tmp/shop-certificate.*", which fails but still does not abort the build - that's problem #2.

[jira-importer]

mrw:

I also tried to specify the source files comma separated, instead of with "*", but that also does not work.

First: Why doesn't it work?
Second: Why doesn't it abort on failure?

[jira-importer]

cadams:

Same problem. I have tested by explicitly exiting with non-zero status and still get:

{{
SSH: EXEC: completed after 23,017 ms
SSH: Disconnecting configuration [centos@] ...
SSH: Transferred 1 file(s)
Finished: SUCCESS
}}

My guess is that it's exiting with the original scp/sftp transfer set RC and not the Exec command status.

[jira-importer]

slide_o_mix:

This would be a change in behavior to the current plugin, I am not sure how many people are relying on the current behavior. I'll have to think about this some more.

[jira-importer]

phreakadelle:

I am encountering simliar issues. I am executing some bash kung-fu on the target server and if this fails, i would like to exit the "step"/plugin with a RC != 0.

The pipeline does not respect that the step has failed and continues the pipeline, so that the following step fails.

[Pipeline] step
SSH: Connecting from host [53d1c1e2cbf8]
SSH: Connecting with configuration [swpsws47] ...
SSH: EXEC: STDOUT/STDERR from command [

    if [ -d /export/yyy/install/xxxmx-soapui/ ]; then
cd /export/yyy/install/xxxmx-soapui/
rm -fv /export/yyy/install/soapui.tar.gz
tar -zcf ../soapui.tar.gz ./*
    else 
echo "This build does not contain SoapUI tests to execute. Please update your branch!"
exit 9
    fi       ] ...
This build does not contain SoapUI tests to execute. Please update your branch!
SSH: EXEC: completed after 201 ms
SSH: Disconnecting configuration [swpsws47] ...
ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]]
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Download Tests)
[Pipeline] sh
[xxx-smoketest-swpsws47] Running shell script
+ wget -q http://swpsws47/yyy/install/soapui.tar.gz
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Verify Server Ready)
Stage "Verify Server Ready" skipped due to earlier failure(s) 

The pipeline looks like

pipeline {
   agent any

   stages {
       stage('Prepare Tests') {
   steps {
       cleanWs notFailBuild: true
       sshPublisher(publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '''

    if [ -d /export/yyy/install/xxxmx-soapui/ ]; then
cd /export/yyy/install/xxxmx-soapui/
rm -fv /export/yyy/install/soapui.tar.gz
tar -zcf ../soapui.tar.gz ./*
    else 
echo "This build does not contain have SoapUI tests to execute. Please update your branch!"
exit 9
    fi       ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
   }
       }

       stage('Download Tests') {
    steps {
       sh 'wget -q http://$ServerName/yyy/install/soapui.tar.gz'
       sh 'tar -xzf soapui.tar.gz --directory .'
    }
       }

       stage('Verify Server Ready') {
       ...
       }

       }
       }

[jira-importer]

phreakadelle:

Here are working example. Just set the variable $ServerName

I would expect that the pipeline does not continue if the sshPublisher plugin returns with something != 0

pipeline {
   agent any

   stages {
       stage('Do sth remote') {
   steps {
       cleanWs notFailBuild: true
       sshPublisher(publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '''
   echo "i am a failure"
   exit 9       ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
   }
       }

       stage('Do sth on jenkins') {
    steps {
       echo "Should not run any more"
    }
       } 
   }
}
 

Outcome:

[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Do sth remote)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] step
SSH: Connecting from host [53d1c1e2cbf8]
SSH: Connecting with configuration [swpsws47] ...
SSH: EXEC: STDOUT/STDERR from command [
   echo "i am a failure"
   exit 9

       ] ...
i am a failure
SSH: EXEC: completed after 201 ms
SSH: Disconnecting configuration [swpsws47] ...
ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]]
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Do sth on jenkins)
[Pipeline] echo
Should not run any more
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE

[jira-importer]

slide_o_mix:

Maybe I can provide an option that will fail the build if there is a non-zero exit status. This would keep the current default, but allow people who want this behavior to have it. I'll look into it.

[jira-importer]

slide_o_mix:

It looks like there is already a flag for this, can you try changing you pipeline as follows:

The change is adding failOnError: true before the publishers list.

pipeline {
   agent any

   stages {
       stage('Do sth remote') {
   steps {
       cleanWs notFailBuild: true
       sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '''
   echo "i am a failure"
   exit 9       ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
   }
       }

       stage('Do sth on jenkins') {
    steps {
       echo "Should not run any more"
    }
       } 
   }
}

[jira-importer]

slide_o_mix:

The thing that needs to be changed is the option to fail the build when no files were copied.

[jira-importer]

phreakadelle:

Even if i change failOnError to true it still executes the next step.

SSH: Connecting from host [3e88a1716d53]
SSH: Connecting with configuration [swpsws46] ...
SSH: EXEC: STDOUT/STDERR from command [
   echo "i am a failure"
   exit 9       ] ...
i am a failure
SSH: EXEC: completed after 201 ms
SSH: Disconnecting configuration [swpsws46] ...
ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]]
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Do sth on jenkins)
[Pipeline] echo
Should not run any more
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: FAILURE

[jira-importer]

phreakadelle:

We can see that there is this Exception thrown but its not aborting the next steps in the pipeline. I am not too deep in the details, but maybe this exception must not be catched?

https://github.com/jenkinsci/publish-over-ssh-plugin/blob/master/src/main/java/jenkins/plugins/publish_over_ssh/BapSshClient.java#L283-L300

private void exec(final BapSshTransfer transfer) {
ChannelExec exec = null;
try {
    exec = openExecChannel();
    exec.setPty(transfer.isUsePty());
    exec.setAgentForwarding(transfer.isUseAgentForwarding());
    exec.setInputStream(null);
    exec.setOutputStream(buildInfo.getListener().getLogger(), true);
    exec.setErrStream(buildInfo.getListener().getLogger(), true);
    connectExecChannel(exec, Util.replaceMacro(transfer.getExecCommand(), buildInfo.getEnvVars()));
    waitForExec(exec, transfer.getExecTimeout());
    final int status = exec.getExitStatus();
    if (status != 0)
throw new BapPublisherException(Messages.exception_exec_exitStatus(status));
} finally {
    disconnectExecQuietly(exec);
}
    }

Ping Alex Earl

[jira-importer]

pinguxx:

I think i have the same problem, pipeline just keeps going, it does mark it as failed but it shouldn't continue the flow, have you found a fix for that Stephan Watermeyer ?

[jira-importer]

phreakadelle:

No. No Solution so far.

The interesting thing is, that the pipeline continues but further sshPublisher steps are canceled with message: SSH: Current build result is [FAILURE], not going to run.

Pipeline

pipeline {
   agent any

   stages {
       stage('Do sth remote') {
   steps {
       cleanWs notFailBuild: true
       sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: '''
   echo "i am a failure"
   exit 9       ''', execTimeout: 18000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: true)])

   }
       }

       stage('Do sth on jenkins') {
    steps {
       echo "Should not run any more"
    }
       }

       stage('a') {
    steps {
echo "before"
sshPublisher(failOnError: true, publishers: [sshPublisherDesc(configName: "$ServerName", transfers: [sshTransfer(excludes: '', execCommand: 'echo $HOSTNAME', execTimeout: 180000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '/export/pce/inbox', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '*.zip')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: true)])
echo "after"
    }
       }
       stage('1') {
    steps {
       sh 'exit 9'
       sh 'echo after exit'
    }
       }

       stage('2') {
    steps {
       sh 'echo finally'
    }
       }
   }
}

Outcome

SH: Connecting from host [f561680303be]
SSH: Connecting with configuration [swpsws46] ...
SSH: Creating session: username [root], hostname [10.226.2.95], port [22]
SSH: Connecting session ...
SSH: Connected
SSH: Opening exec channel ...
SSH: EXEC: channel open
SSH: EXEC: STDOUT/STDERR from command [
   echo "i am a failure"
   exit 9       ] ...
SSH: EXEC: connected
i am a failure
ERROR: Exception when publishing, exception message [Exec exit status not zero. Status [9]]
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Do sth on jenkins)
[Pipeline] echo
Should not run any more
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (a)
[Pipeline] echo
before
[Pipeline] step
SSH: Current build result is [FAILURE], not going to run.
[Pipeline] echo
after
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (1)
[Pipeline] sh
+ exit 9
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (2)
Stage "2" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 9
Finished: FAILURE

I don't know how and where to fix this. I would assume that somewhere a AbortException or something must be raised, but the current Interface for the endTransfer Method does not allow such modifications.

This is something for a more experienced guy

[jira-importer]

pinguxx:

Since the plugin mark the build as failure, you could check if the result is failure after it ran

 

if ('FAILURE'.equals(currentBuild.result)) {
    error('ssh error')
}

 

[lainosantos]

Hello, any fix for this?

[ernitingarg]

Hello, any fix for this, please?