infra errors during DIB setup + virt-customize are not caught automatically

[halcyondude]

When rpm's are not available, or packaging conflicts occur during yum install portions of image build, they silently go into either /tmp/builder.log in the libguestfs appliance, or to the output of one of the later phases. The role then typically fails later, when either the image is not constructed correctly or simply does not exist (primary scenario).

This incurs a manual debug / investigation of the logs to find the culprit. This issue is to track the addition of either more intelligent log auto-scrubbing than we do now (ample opportunity for improvement) and/or earlier failure where possible (e.g., if an error happened during setup of DIB and build host, don't run dib itself, only to fail later.

---

A concrete example: Adding new version of Ceph for newton:

• https://review.gerrithub.io/#/c/286365/
• failed gate job: https://ci.centos.org/job/artib-gate-mitaka-delorean-full-images/39/

Root cause was that the dib setup script was making ref to a bad URI

• https://ci.centos.org/artifacts/rdo/jenkins-artib-gate-mitaka-delorean-full-images-39/172.19.2.22/var/log/artib/buildhost.dib-prepare-script.sh.log.gz

and the following error was being silently ignored:

• yum install -y https://trunk.rdoproject.org/centos7-master/29/d6/29d6f749d7bab01fa3defb0c1c6e1f95b268d665_4dec0972/tripleo-common-2.0.1-0.20160503234526.29d6f74.el7.centos.noarch.rpm Cannot open: https://trunk.rdoproject.org/centos7-master/29/d6/29d6f749d7bab01fa3defb0c1c6e1f95b268d665_4dec0972/tripleo-common-2.0.1-0.20160503234526.29d6f74.el7.centos.noarch.rpm. Skipping. Error: Nothing to do

@weshayutin FYI

[halcyondude]

Another concrete example, this time when setting up the repos in the base image. From #25 (closed as dupe)

• https://ci.centos.org/job/artib-gate-liberty-delorean-full-images/33/
• https://ci.centos.org/artifacts/rdo/jenkins-artib-gate-liberty-delorean-full-images-33/172.19.2.67/var/log/artib/repo_setup.sh.log.gz

The build host could not get to repos, the repo_setup.sh step/script failed, the build then failed, but the job then went on to attempt to test (because we swallowed the error in artib).