docker / buildx

Docker CLI plugin for extended build capabilities with BuildKit
Apache License 2.0
3.57k stars 483 forks source link

Inside buildx created QEMU image, /proc/self/exe does not refer to the same device/inode as the actual executable #1331

Open shabiel opened 2 years ago

shabiel commented 2 years ago

Hello everybody.

Thank you for your help in advance.

We are an open source project, and we are exploring using buildx to automatically create ARM images for our database product from x86 runners on Gitlab.

We found a pretty strange issue. Our database is daemonless, and we have code to check that the database executable running indeed refers to itself. We have different code for different platforms (AIX, Linux, Tru64). On Linux, we use stat(2) and we compare the stat members st_dev and st_ino. This works well in bare-metal ARM machines, and forever practically on x86 architectures (the oldest release in our git repo, which is > 20 years old, has the same code).

Here's some debugging output:

amd64:

image: /home/sam/work/gitlab/YDB/build/yottadb
real_dist: /home/sam/work/gitlab/YDB/build
exename: yottadb
image_real_path: /proc/self/exe
comparison: /home/sam/work/gitlab/YDB/build/yottadb
ID of containing device1:  [fd,8]
ID of containing device2:  [fd,8]
I-node number1:            109523586
I-node number2:            109523586

arm64 on buildx:

image: /YDB/build/yottadb
real_dist: /YDB/build
exename: yottadb
image_real_path: /proc/self/exe
comparison: /YDB/build/yottadb
ID of containing device1:  [0,4e]
ID of containing device2:  [0,50]
I-node number1:            1849208
I-node number2:            2786994

Here's the code the performs the comparison: https://gitlab.com/shabiel/YDB/-/blob/arm64-docker-images/sr_unix/is_file_identical.c

tonistiigi commented 2 years ago

There are known issues in qemu on handling /proc/self/exe. Some example unmerged patches https://patchwork.kernel.org/project/qemu-devel/patch/20210531055019.10149-9-yamamoto@midokura.com/