search

dotnet / dotnet-buildtools-prereqs-docker

Used to maintain the Docker images hosted at the mcr.microsoft.com/dotnet-buildtools/prereqs image repository
MIT License
55 stars 103 forks source link

Ubuntu 16.04 Cross image unable to build CoreCLR on ARM32 #120

Closed jkoritzinsky closed 5 years ago

jkoritzinsky commented 5 years ago

In dotnet/coreclr#23999 I tried to update the docker image that we use to build CoreCLR on Linux ARM to be based on Ubuntu 16.04 instead of 14.04 so we can move off 14.04 before it EOLs. For whatever reason, the build with the 16.04 image fails. I don't know enough about how our Docker images work to efficiently debug this issue myself. Here's a link to the failed build on Azure DevOps: https://dev.azure.com/dnceng/public/_build/results?buildId=156148&view=logs&jobId=d37b6b50-5a2d-5da1-80ef-d0fdb8e5bff2

From my first glance it seems that the pre-build hook might not be getting run correctly. Does that seem possible?

MichaelSimons commented 5 years ago

I took a look the the image you are using and something does seem fishy as the /crossrootfs/arm directory is only 129MB vs 810MB in the 14.04 image. While I don't have the build logs for the image you are referencing, I took a look at the latest PR validation and the build-rootfs.sh output seems suspicious.

Line 11651 of the Ubuntu 16.04 leg - https://dev.azure.com/dnceng/public/_build/results?buildId=153775

I: Running command: chroot /rootfs/arm /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'symlinks': No such file or directory
Checking existence of /rootfs/arm/bin
Cleaning apt files
Tarring rootfs
Shutting down container
rootfs-arm-xenial-1555104764

IIRC there have been previous issues in the past with the script silently failing.

+@jashook who has worked on this in the past.

MichaelSimons commented 5 years ago

I kicked off an official build and it shows the same symptoms - https://dev.azure.com/dnceng/internal/_build/results?buildId=161396

mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-406629a-20190419022644

MichaelSimons commented 5 years ago

@jashook - Is this something you can look at or recommend who should investigate?

jashook commented 5 years ago

@MichaelSimons I will not be able to get to this for a bit. /cc @RussKeldorph

wfurt commented 5 years ago

I can probably take a look but it may be few days. From the log, it seems like we are missing qemu-static in base image. 

chroot: failed to run command '/debootstrap/debootstrap': Exec format error

That is part of magic allowing to run ARM binaries on x64 machine.

janvorli commented 5 years ago

I am going to look into fixing it.

janvorli commented 5 years ago

There is something strange going on. When I build the images from src/ubuntu/14.04/coredeps, then the src/ubuntu/14.04/crossdeps and run the src/ubuntu/14.04/cross/hooks/pre-build script on my local Ubuntu 18.04 VM, it successfully builds the arm rootfs that contains all the necessary files. Since that runs in a container, it is not clear to me where the issue stems from. @MichaelSimons what is the distro we run that on in the official build?

MichaelSimons commented 5 years ago

Ubuntu 16.04 host - https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/master/.vsts-pipelines/stages/build-repo.yml#L21

janvorli commented 5 years ago

I have found someone else having the same issue: https://github.com/RPi-Distro/pi-gen/issues/133. My guess is that the host needs to have the binfmt-support installed even though we are building our stuff in docker container. I am going to install a clean Linux 16.04 and give that theory a try.

janvorli commented 5 years ago

Ok, I've confirmed that I got the same failure on a fresh installation of Linux 16.04 and that the issue goes away after installing binfmt-support, qemu and qemu-user-static. I'm not sure yet if all of the three were needed, I've installed them in this specific order and it fixed the problem after the qemu-user-static was installed. I'll do a little more testing to see if only part of those are needed.

MichaelSimons commented 5 years ago

It is certainly possible for us to add these to the build agent. It will have to happen during each build since we are utilizing hosted agents.

janvorli commented 5 years ago

I've verified that all of the three packages are needed. Removing any of them results in the issue coming back.

MichaelSimons commented 5 years ago

Upon thinking about this, I think the best place to put this logic is in the build-rootfs.sh script in this repo.

janvorli commented 5 years ago

@MichaelSimons I assume this script is always executed with sudo / under a privileged account, otherwise we would not be able to install packages. Is that correct? Do you want me to add the line to install the three packages or would you do that?

MichaelSimons commented 5 years ago

I was going to have @mthalman do this but we are both traveling today. I don't recall how the script is invoked.

janvorli commented 5 years ago

OK, I'll leave this change to @mthalman then once he's back.

mthalman commented 5 years ago

I've fixed the build pipeline to resolve these errors that were showing up:

I: Running command: chroot /rootfs/arm /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'apt-get': No such file or directory
chroot: failed to run command 'symlinks': No such file or directory
Checking existence of /rootfs/arm/bin
Cleaning apt files
Tarring rootfs
Shutting down container
rootfs-arm-xenial-1555104764

These are the images produced by this fix: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-cross-cfdd435-20190520220751 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-3e800f1-20190520220755 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-debpkg-cfdd435-20190520220755 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-cfdd435-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-alpine-406629a-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-406629a-20190520220848 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-09ca40b-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-debpkg-cfdd435-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-mlnet-207e097-20190520220842 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-arm64v8-3e800f1-20190520220748 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-09ca40b-20190520220751 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-arm32v7-0dbbad8-20190520220749 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-arm32v7-0dbbad8-20190520220749 mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm32v7-09ca40b-20190520220745