[wip] Maybe qemu 4.2 will fix armhf and arm64 Focal Nbin job failures

[sloretz]

This is a work in progress PR I'm exploring as a solution to the Noetic Focal armhf and arm64 job failures that started 2 days ago. I'll push just osrf/ubuntu_armhf:focal and see if it fixes the failures for Noetic.

I'm trying a newer version of qemu, though I have little reason to think it will help. I notice the latest version of qemu has some changes in do_semop() compared to 3.1. Maybe something on the build agent's changed, like the kernel version, and a newer qemu version is able to handle this new case?

• Example successful job on March 3
• Same job failing on March 18

Random notes about the failure

I don't have a better place to put these at the moment. Maybe I'll move this to a ticket somewhere else when I have an opinion about where that ticket should live.

• Only armhf and arm64 jobs on Ubuntu Focal are failing
  • Focal amd64, Buster amd64, and Buster arm64 jobs are not failing
• The last successful job was 6 days ago, and the first failure was 2 days ago, so it must be from a change after March 13th but before March 17th.
• For all failing jobs, it shows in the console as:

  dpkg-source: info: using options from ros-noetic-pcl-msgs-0.3.0/debian/source/options: --auto-commit
  fakeroot debian/rules clean
  semop(1): encountered an error: Function not implemented
  dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 1
  E: Building failed
  Traceback (most recent call last):
  File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 138, in build_binarydeb
  subprocess.check_call(cmd, cwd=source_dir)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
  raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-noetic-pcl-msgs']' returned non-zero exit status 1.

• The error happens when fakeroot calls the syscall semop() to decrement a semaphore
• The error Function not implemented is what perror() prints when a syscall returns ENOSYS
• ENOSYS is not one of the documented error codes for semop()
• qemu seems to translate syscalls, including semop(), though I wasn't able to find a spot where it would return ENOSYS in the stable-3.1 branch.
• Both old successful and new failing jobs use the same build of osrf/ubuntu_armhf:focal and osrf_ubuntu_arm64:focal, meaning they have exactly the same version of qemu, so that hasn't changed.
• Both old successful and new failing jobs use the same version of fakeroot, 1.24-1
• Both old successful and new failing jobs use the exact same version of Docker, though I'm not sure how involved it would be in syscalls since the container uses the host's kernel.

• docker version Client: Docker Engine - Community Version: 19.03.2 API version: 1.40 Go version: go1.12.8 Git commit: 6a30dfc Built: Thu Aug 29 05:28:19 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.2 API version: 1.40 (minimum version 1.12) Go version: go1.12.8 Git commit: 6a30dfc Built: Thu Aug 29 05:26:54 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.6 GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc: Version: 1.0.0-rc8 GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f docker-init: Version: 0.18.0 GitCommit: fec3683

* @clalancette says the path to investigate is `fakeroot`-> container `glibc` -> `qemu` -> host `glibc` -> `kernel`
* Successful job had container `glibc` version `2.30-0ubuntu3`
* Failing job had container `glibc` version `2.31-0ubuntu6`

[clalancette]

• ENOSYS is not one of the documented error codes for semop()

I poked at the glibc sources just a bit this morning. It looks like there are two ways glibc can return ENOSYS from semop: either if it thinks the kernel isn't Linux, or if the kernel itself returns ENOSYS. I couldn't find any other ways this could happen.

• qemu seems to translate syscalls, including semop(), though I wasn't able to find a spot where it would return ENOSYS in the stable-3.1 branch.

Not directly, but these are again translated through glibc and then through to the kernel. Also, I know this may not help directly, but in the master branch of qemu, do_semop can return ENOSYS if semtimedop isn't recognized as a valid syscall.

You can also use the perf tool to try and get an idea if these syscalls are hitting the kernel and returning ENOSYS there. Running something like:

perf trace -e semop

Will tell you about all of the semop syscalls that are happening on the system. If you know the PID of the process you want to trace, you can also add -p <pid>, which will restrict it just to that PID.

I'm not sure if any of these ramblings are helpful, but just some other ideas for you.

[sloretz]

FWIW, upgrading qemu to 4.2 did not fix the issue: http://build.ros.org/view/Nbin_ufhf_uFhf/job/Nbin_ufhf_uFhf__rosbash__ubuntu_focal_armhf__binary/9/

[sloretz]

Closing since it did not fix the issue. Notes moved to #36