Bionic builds for Arm{64,hf} sometimes fail with a corrupt package

[clalancette]

Starting around April 27th, we've had a number of incidents where Armhf or Arm64 builds for Melodic on Bionic have failed with a "corrupt" humanity-icon-theme package (example). If we look closely at that log, at timestamp 23:05:03, we see the following:

23:05:03 Preparing to unpack .../041-humanity-icon-theme_0.6.15_all.deb ...
23:05:03 Unpacking humanity-icon-theme (0.6.15) ...
23:05:05 dpkg: error processing archive /tmp/apt-dpkg-install-nKKgiP/041-humanity-icon-theme_0.6.15_all.deb (--unpack):
23:05:05  corrupted filesystem tarfile - corrupted package archive

(this then causes cascading failures later on). Note that this failure is random; running the same build again may make this succeed. I spent some time with @nuclearsandwich debugging this on one of the ros buildfarm hosts, and it looks like the package is being properly downloaded (I confirmed by doing an md5sum against the downloaded package and verifying that it matched the upstream one). But it seems like during uncompression it sometimes gets corrupted. I'll also note that when the failure happens it always happens because of this package; I've never seen any others.

Here are some links to additional failures, though this is not an exhaustive list:

• http://build.ros.org/job/Mbin_ubv8_uBv8__tf__ubuntu_bionic_arm64__binary/7/consoleFull
• http://build.ros.org/job/Mbin_ubv8_uBv8__rqt_image_view__ubuntu_bionic_arm64__binary/3/consoleFull
• http://build.ros.org/job/Mbin_ubv8_uBv8__rqt_image_view__ubuntu_bionic_arm64__binary/2/consoleFull
• http://build.ros.org/job/Mbin_ubv8_uBv8__stage__ubuntu_bionic_arm64__binary/9/consoleFull
• http://build.ros.org/job/Mbin_ubhf_uBhf__turtle_tf2__ubuntu_bionic_armhf__binary/7/consoleFull

[nuclearsandwich]

Some notes from continued investigation. This has proved extremely heisenbuggy.

Ways we've been able to reproduce the issue:

• Running the job on production jenkins agents
• Running the release script on production job agents (occasionally succeeds)

Ways we've been unable to reproduce the issue:

• Running a docker container with docker run -ti osrf/ubuntu_arm64:bionic bash and installing the package by hand on workstations or production agents
• Running the install command on a production agent outside of a docker build step
• Running apt-get install humanity-icon-theme in a container matching the pre-installation step of the build script on a production agent. *Interrupting the docker image build one step before installation and running the install command manually on a production agent.

Possible workarounds

• Pre-install humanity-icon-theme on osrf/ubuntu_arm{64,hf}:bionic
• Updating docker to 18.04 We have not been able to reproduce in 18.04 yet but we also haven't used it during a Jenkins job.

[nuclearsandwich]

Docker 18.03 has been deployed. I'm going to monitor for a few days at least before determining for sure whether or not we can stay.