Value too large for defined data type when cargo build on armv7 with qemu

[jdrouet]

Problem

Trying to build an app for armv7 in a docker container with qemu. The current result is explained here with the steps to reproduce it : https://github.com/jdrouet/docker-bug-value-too-long

Steps

1. install docker
2. install docker-buildx
3. follow the steps here

Possible Solution(s) No idea.

Notes The guys from docker claim it's coming from cargo.

Output of cargo version:

The one from the docker image rust:1-slim-buster

[ehuss]

Thanks for the report. I think this is a duplicate of #7451. There is some sort of strange interaction of Linux Large File Support with a 64-bit host and a 32-bit docker image running qemu. There's some more discussion on that issue, but in particular see https://bugs.launchpad.net/qemu/+bug/1805913. I'm not sure there is anything Cargo can do about this. I suspect the underlying problem is that libgit2 uses size-agnostic functions like readdir instead of explicitly using readdir64. It may be possible to build libgit2 with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 (or getconf LFS_CFLAGS), but I have no idea if that will work or what the consequences would be (it seems like it could have portability issues). This may also just be an issue with qemu.

[diabloxenon]

Hello all,

I have a similar use case i.e. (Qemu userspace emulation for armv7 on x86_64 arch). After chrooting inside the image, i tried building my project repository using that and this was the output.

[root@mnt ~]# cargo build --target=armv7-unknown-linux-gnueabihf
    Updating crates.io index
warning: spurious network error (2 tries remaining): could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
warning: spurious network error (1 tries remaining): could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  could not read directory '/root/.cargo/registry/index/github.com-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
[root@mnt ~]# 

However, when fetching it manually using curl https://github.com/rust-lang/crates.io-index from chroot, this worked exactly as it should have. Because of that, I also tested this on qemu aarch64 image and there it worked perfectly.

As suggested earlier by @ehuss, it seems the issue is still persistent to this date with qemu emulation of armv7. The cargo binaries are definitely not the culprit here, and more likely libgit2 seems to be causing this issue.

Internally, there is some incompatiblity going on, even after following the suggestion for building the library with parameters provided by getconf LFS_CFLAGS, the issue remains the same.

Lastly, for this quick and simple test case, I would prefer to use command cargo install exa inside an emulated armv7 image already installled with Rust, for any future testing regarding to its compatiblity.

EDIT: I also proposed a simple solution for anyone that is currenly facing this problem, and until its resolution. #9545

[tillsteinbach]

If someone is coming here and also needs a workaround for a docker build: Put this in the beginning of the Dockerfile: # syntax = docker/dockerfile:experimental Then do something like this: RUN --security=insecure mkdir -p /root/.cargo && chmod 777 /root/.cargo && mount -t tmpfs none /root/.cargo && some_command_that_uses_cargo

[nijel]

If you can't make the above Dockerfile-based workaround, what works as well is using tmpfs for the whole /var/lib/docker on the build host. For GitHub actions, I've done that here: https://github.com/WeblateOrg/docker/pull/1274

[JonasAlfredsson]

Very nice @nijel! I used your example and made a reusable Action for those affected by this issue :)

[SuperSandro2000]

If someone is coming here and also needs a workaround for a docker build: Put this in the beginning of the Dockerfile: # syntax = docker/dockerfile:experimental Then do something like this: RUN --security=insecure mkdir -p /root/.cargo && chmod 777 /root/.cargo && mount -t tmpfs none /root/.cargo && some_command_that_uses_cargo

If you are using buildx you also need to allow insecure-entitlement when creating the node:

docker buildx create --driver docker-container --name local --buildkitd-flags '--allow-insecure-entitlement security.insecure' --use

https://github.com/docker/buildx/issues/220#issuecomment-574377254

and allow insecure things in the docker daemon:

# cat /etc/docker/daemon.json
{ "builder": {"Entitlements": {"security-insecure": true }}}

https://github.com/docker/buildx/issues/559#issuecomment-796430825

[djmaze]

If you are using buildkit, there is a much cleaner workaround than using the --security=insecureoption. Just use RUN --mount like this:

RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo

[tropxy]

If you are using buildkit, there is a much cleaner workaround than using the --security=insecureoption. Just use RUN --mount like this:

RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo

For me this worked, but I had to install rust within the same RUN layer:

RUN --mount=type=tmpfs,target=/root/.cargo curl https://sh.rustup.rs -sSf | bash -s -- -y && pip install cryptography==38.0.1

finally, to activate Buildkit, I had to run docker build as:

DOCKER_BUILDKIT=1 docker build .

[philw07]

If you are using buildkit, there is a much cleaner workaround than using the --security=insecureoption. Just use RUN --mount like this:

RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo

I had to adapt this slightly to use a different directory for cargo to get it working with the rust base image to build a rust project.

RUN --mount=type=tmpfs,target=/.cargo CARGO_HOME=/.cargo cargo build --release

[flo-at]

Doesn't the tmpfs workaround hurt caching a lot? Cargo has to download and build everything every time if CARGO_HOME is not stored on the filesystem.

[omarandlorraine]

Notes The guys from docker claim it's coming from cargo.

It's a problem with the virtualization layer. The error is not coming from cargo or docker, but from QEMU running 32-bit platforms on 64-bit hosts (I don't mean it's a bug in QEMU though). An alternative workaround could be to run a 32-bit QEMU process if you have a multilib set up. This would mean using the host kernel's 32-bit API, which does not suffer from the conversion problem.

[flo-at]

Since cargo (stable) now supports the sparse registry protocol, the easiest workaround is to set the envinronment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse. This avoids the problematic call to libgit2 altogether. You have to make sure the git binary is installed though, but that shouldn't be a problem.

[MichaIng]

@flo-at CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse works very well, many thanks. However, basically the same error happens on rustup self uninstall and when temporary directories are attempted to be removed during the process:

info: syncing channel updates for '1.73.0-armv7-unknown-linux-gnueabihf'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
warning: could not delete temp directory: /root/.rustup/tmp/dq_vgjwah9211lz1_dir
info: installing component 'clippy'
warning: could not delete temp directory: /root/.rustup/tmp/_42_3qtwdsd2gjit_dir
info: installing component 'rust-std'
warning: could not delete temp directory: /root/.rustup/tmp/hg7ojeltqnog7xdr_dir
info: installing component 'rustc'
warning: could not delete temp directory: /root/.rustup/tmp/a10n8s4a7catp38h_dir
info: installing component 'rustfmt'
warning: could not delete temp directory: /root/.rustup/tmp/8fbaomir76h2yb8j_dir
    Updating crates.io index
 Downloading crates ...
...
# rustup self uninstall -y
info: removing rustup home
error: could not remove 'rustup_home' directory: '/root/.rustup': Value too large for defined data type (os error 75)

Are there similar options for cargo and rustup past the installation? Otherwise to me the tmpfs workaround seems cleaner, to avoid possible surprises.

[madiele]

For anyone having this issue, in the end I went to the path of making a Dockerfile with multi-stage builds that will cross-compile for all the architectures I need on the architecture of the pipeline agent and scrapped using qemu for the cargo build, I only use it to run apt install and other small stuff at the end.

If you project allows for this route, you can follow my Dockerfile as guidance

https://github.com/madiele/vod2pod-rss/blob/cac3d27c415ed8341fa8da738e4604b7427c04ae/Dockerfile

I went form 2+ hours build times to 5-10 minutes build times, it was worth the effort.

[chairwa]

docker run --privileged -itd --network host --mount type=tmpfs,target=/root/.cargo arm32v7/ubuntu this way, /root/.cargo/bin/cargo bash: ./cargo: Permission denied