Huge performance difference for multi arch container builds between running buildkitd on the host or inside a container

[akhal3d96]

I'm trying to build an arm64 container image on an amd64 virtual machine that supports nested virtualization.

I have Docker on this machine so I run buildkit v0.14.0 with privileged mode on the host and used buildctl on the same host to build the container image but it was too slow (took hours to finish). I also tried to mount /dev/kvm inside buildkit container but it didn't change anything. I tried running buildkitd directly on the host and tried to build the same image with buildctl and it finished within minutes.

I'm suspecting that there's something in the container runtime blocking the qemu inside buildkit from accessing the nested virtualization capabilities but I can't trace it.

docker version 26.1.1
buildkit v0.14.0

I'd appreciate your help and thanks in advance.

[tonistiigi]

Buildkit does not create VMs or use KVM or /dev/kvm anywhere. It isn't clear what setups you are using, what build commands invoking or where you see the slowness.

[akhal3d96]

When I'm building the container image, it takes so much time on this step when I run buildkit inside a container:

RUN dnf install glibc-langpack-en

Exactly when it's running the post installation hooks:

=> => #   Running scriptlet: glibc-2.34-100.0.1.el9_4.2.aarch64                  19/148

Which are, AFAIK, bunch of bash and lua scripts.

I build the Dockerfile using buildctl in the two cases:

buildctl --addr tcp://0.0.0.0:1468 build --opt platform=linux/arm64 --frontend dockerfile.v0 --local context=. --local dockerfile=. --opt build-arg:TARGETARCH=arm64 --opt build-arg:TARGETPLATFORM=linux/arm64 --output type=image,name=docker.io/username/image

When I'm running buildkit inside a container:

# Very slow while buildinh arm64
docker run --name buildkitd --network=host --privileged moby/buildkit:v0.14.1

This is how I run it outside the container:

# Building arm64 containers is fast  
# I run it as root
./buildkitd --addr tcp://0.0.0.0:1468