Closed jkoritzinsky closed 5 years ago
I took a look the the image you are using and something does seem fishy as the /crossrootfs/arm
directory is only 129MB vs 810MB in the 14.04 image. While I don't have the build logs for the image you are referencing, I took a look at the latest PR validation and the build-rootfs.sh
output seems suspicious.
Line 11651 of the Ubuntu 16.04 leg - https://dev.azure.com/dnceng/public/_build/results?buildId=153775
I: Running command: chroot /rootfs/arm /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'symlinks': No such file or directory
Checking existence of /rootfs/arm/bin
Cleaning apt files
Tarring rootfs
Shutting down container
rootfs-arm-xenial-1555104764
IIRC there have been previous issues in the past with the script silently failing.
+@jashook who has worked on this in the past.
I kicked off an official build and it shows the same symptoms - https://dev.azure.com/dnceng/internal/_build/results?buildId=161396
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-406629a-20190419022644
@jashook - Is this something you can look at or recommend who should investigate?
@MichaelSimons I will not be able to get to this for a bit. /cc @RussKeldorph
I can probably take a look but it may be few days. From the log, it seems like we are missing qemu-static in base image.
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
That is part of magic allowing to run ARM binaries on x64 machine.
I am going to look into fixing it.
There is something strange going on. When I build the images from src/ubuntu/14.04/coredeps, then the src/ubuntu/14.04/crossdeps and run the src/ubuntu/14.04/cross/hooks/pre-build script on my local Ubuntu 18.04 VM, it successfully builds the arm rootfs that contains all the necessary files. Since that runs in a container, it is not clear to me where the issue stems from. @MichaelSimons what is the distro we run that on in the official build?
I have found someone else having the same issue: https://github.com/RPi-Distro/pi-gen/issues/133. My guess is that the host needs to have the binfmt-support installed even though we are building our stuff in docker container. I am going to install a clean Linux 16.04 and give that theory a try.
Ok, I've confirmed that I got the same failure on a fresh installation of Linux 16.04 and that the issue goes away after installing binfmt-support, qemu and qemu-user-static. I'm not sure yet if all of the three were needed, I've installed them in this specific order and it fixed the problem after the qemu-user-static was installed. I'll do a little more testing to see if only part of those are needed.
It is certainly possible for us to add these to the build agent. It will have to happen during each build since we are utilizing hosted agents.
I've verified that all of the three packages are needed. Removing any of them results in the issue coming back.
Upon thinking about this, I think the best place to put this logic is in the build-rootfs.sh script in this repo.
@MichaelSimons I assume this script is always executed with sudo / under a privileged account, otherwise we would not be able to install packages. Is that correct? Do you want me to add the line to install the three packages or would you do that?
I was going to have @mthalman do this but we are both traveling today. I don't recall how the script is invoked.
OK, I'll leave this change to @mthalman then once he's back.
I've fixed the build pipeline to resolve these errors that were showing up:
I: Running command: chroot /rootfs/arm /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'symlinks': No such file or directory
Checking existence of /rootfs/arm/bin
Cleaning apt files
Tarring rootfs
Shutting down container
rootfs-arm-xenial-1555104764
These are the images produced by this fix: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-cross-cfdd435-20190520220751 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-3e800f1-20190520220755 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-debpkg-cfdd435-20190520220755 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-cfdd435-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-alpine-406629a-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-406629a-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-09ca40b-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-debpkg-cfdd435-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-mlnet-207e097-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-arm64v8-3e800f1-20190520220748 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-09ca40b-20190520220751 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-arm32v7-0dbbad8-20190520220749 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-arm32v7-0dbbad8-20190520220749 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm32v7-09ca40b-20190520220745
In dotnet/coreclr#23999 I tried to update the docker image that we use to build CoreCLR on Linux ARM to be based on Ubuntu 16.04 instead of 14.04 so we can move off 14.04 before it EOLs. For whatever reason, the build with the 16.04 image fails. I don't know enough about how our Docker images work to efficiently debug this issue myself. Here's a link to the failed build on Azure DevOps: https://dev.azure.com/dnceng/public/_build/results?buildId=156148&view=logs&jobId=d37b6b50-5a2d-5da1-80ef-d0fdb8e5bff2
From my first glance it seems that the
pre-build
hook might not be getting run correctly. Does that seem possible?