task peadm::pe_install doesn't handle commands exit codes properly [SOLARCH-545]

[vchepkov]

Seems I have started upgrading too early, file hasn't been properly uploaded on yet, so I got this content (consistently)

$ cat /opt/staging/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz 
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>released/2019.5.0/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz</Key><RequestId>281FD5D76966A802</RequestId><HostId>30Us5WIkg+ITt5JFSwxlX184ncmHJH/vWG5lltbLM7NP89F/VSEk2ZwaA/hrb9mfBcikeJbstBE=</HostId></Error>

upgrade plan proceeded all the way to task peadm::pe_install and it executed successfully as well, without doing an upgrade.

debugging of the task showed that results of the tar command are ignored:

$ bolt task run peadm::pe_install --targets puppet.chepkov.lan tarball=/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz puppet_service_ensure=stopped --debug
Loaded configuration from: '/Users/vvc/Bolt/home/bolt.yaml'
Loading modules from /opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bolt-2.2.0/bolt-modules:/Users/vvc/Bolt/home/modules:/Users/vvc/Bolt/home/site-modules:/Users/vvc/Bolt/home/site:/opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bolt-2.2.0/modules
Analytics opt-out is set, analytics will be disabled
Loaded inventory from /Users/vvc/Bolt/home/inventory.yaml
Skipping submission of 'task_run' screenview because analytics is disabled
Started with 100 max thread(s)
Starting: task peadm::pe_install on puppet.chepkov.lan
Skipping submission of 'Transport initialize' event because analytics is disabled
Running task peadm::pe_install with '{"tarball":"/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz","puppet_service_ensure":"stopped","_task":"peadm::pe_install"}' on ["puppet.chepkov.lan"]
Running task run '#<Bolt::Task:0x00007fb040c06618>' on puppet.chepkov.lan
Started on puppet.chepkov.lan...
Initializing ssh connection to puppet.chepkov.lan
Opened session
Running '/Users/vvc/Bolt/home/modules/peadm/tasks/pe_install.sh' with {"tarball":"/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz","puppet_service_ensure":"stopped","_task":"peadm::pe_install"}
Executing: mkdir -m 700 /tmp/645c830d-bc50-44bc-af85-71df61a05759
Command returned successfully
Uploading /Users/vvc/Bolt/home/modules/peadm/tasks/pe_install.sh, to /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh
Executing: chmod u\+x /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh
Command returned successfully
Executing: id -g root
stdout: 0
Command returned successfully
Executing: sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  sh -c 'cd && chown -R root:0 /tmp/645c830d-bc50-44bc-af85-71df61a05759'
Command returned successfully
Executing: PT_tarball=/tmp/puppet-enterprise-2019.5.0-el-8-x86_64.tar.gz PT_puppet_service_ensure=stopped PT__task=peadm::pe_install sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  -E sh -c 'cd && /tmp/645c830d-bc50-44bc-af85-71df61a05759/pe_install.sh'
stderr: tar: This does not look like a tar archive
stderr: gzip: stdin: not in gzip format
stderr: tar: Child returned status 1
tar: Error is not recoverable: exiting now
stderr: dirname: missing operand
Try 'dirname --help' for more information.
stderr: gzip: stdin: not in gzip format
stderr: tar: Child returned status 1
tar: Error is not recoverable: exiting now
stderr: 
stderr: /bin/bash: /tmp//puppet-enterprise-installer: No such file or directory
Command returned successfully
Executing: sudo -S -H -u root -p \[sudo\]\ Bolt\ needs\ to\ run\ as\ another\ user,\ password:\  sh -c 'cd && rm -rf /tmp/645c830d-bc50-44bc-af85-71df61a05759'
Command returned successfully
Closed session
{"target":"puppet.chepkov.lan","action":"task","object":"peadm::pe_install","status":"success","value":{"_output":""}}
Finished on puppet.chepkov.lan:
  Task completed successfully with no result
Finished: task peadm::pe_install with 0 failures in 1.3 sec
Successful on 1 target: puppet.chepkov.lan
Ran on 1 target in 1.37 sec

[reidmv]

This is probably resolved by #75

[vchepkov]

Ideally it would fail a bit earlier, but it does fail, so I think we can close this

Starting: plan peadm::upgrade
Starting: task peadm::trusted_facts on puppet.chepkov.lan
Finished: task peadm::trusted_facts with 0 failures in 3.59 sec
Starting: task peadm::precheck on puppet.chepkov.lan
Finished: task peadm::precheck with 0 failures in 1.28 sec
Starting: plan peadm::util::retrieve_and_upload
Starting: task peadm::download on local://localhost
Finished: task peadm::download with 0 failures in 0.43 sec
Starting: task peadm::filesize on local://localhost
Finished: task peadm::filesize with 0 failures in 0.11 sec
Starting: task peadm::filesize on puppet.chepkov.lan
Finished: task peadm::filesize with 0 failures in 1.62 sec
Starting: file upload from /opt/staging/puppet-enterprise-2020.1.0-el-8-x86_64.tar.gz to /tmp/puppet-enterprise-2020.1.0-el-8-x86_64.tar.gz on puppet.chepkov.lan
Finished: file upload from /opt/staging/puppet-enterprise-2020.1.0-el-8-x86_64.tar.gz to /tmp/puppet-enterprise-2020.1.0-el-8-x86_64.tar.gz with 0 failures in 1.24 sec
Finished: plan peadm::util::retrieve_and_upload in 3.43 sec
Starting: task service on puppet.chepkov.lan
Finished: task service with 0 failures in 2.61 sec
Starting: task peadm::pe_install on puppet.chepkov.lan
Finished: task peadm::pe_install with 1 failure in 1.36 sec
Finished: plan peadm::upgrade in 12.34 sec
Finished: plan maint::upgrade in 16.32 sec
Failed on puppet.chepkov.lan:
  The task failed with exit code 127:
  tar: This does not look like a tar archive

  gzip: stdin: not in gzip format
  tar: Child returned status 1
  tar: Error is not recoverable: exiting now
  dirname: missing operand
  Try 'dirname --help' for more information.

  gzip: stdin: not in gzip format
  tar: Child returned status 1
  tar: Error is not recoverable: exiting now
  /bin/bash: /tmp//puppet-enterprise-installer: No such file or directory
Failed on 1 target: puppet.chepkov.lan
Ran on 1 target

[reidmv]

It might be sufficient to just set -e at the top of the bash script. I haven't tested or considered if there are any intermediate commands that are expected to return non-zero exit codes in normal circumstances. If we were able to use set -e, it would fail immediately as soon as any command failed.

[vchepkov]

yep, that could be the proper solution.

Unfortunately, cloudfront returns 200 even when you download a garbage instead of archive, not much can be done here

[reidmv]

This experience should now be further improved by error handling added to peadm::download in #92.

[davidsandilands]

Fixed by #152