Open nuket opened 1 year ago
Emilgardis
commented
1 year ago Can you please input the details of the error into the description instead of having it in another issue? Thanks!
what virtual filesystem does docker end up using? Does it work better when using cross from the main branch? cargo install cross --git https://github.com/cross-rs/cross
nuket
commented
1 year ago Sure. Docker is using overlay2. I tried this all again using the cross main branch and it gives the same error.
Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#1511a287)` (executables `cross`, `cross-util`) rustuser@rust-build:~/rustuser/hello-world$ cross build --target aarch64-unknown-linux-gnu info: downloading component 'rust-std' for 'aarch64-unknown-linux-gnu'
info: installing component 'rust-std' for 'aarch64-unknown-linux-gnu'
32.8 MiB / 32.8 MiB (100 %) 19.5 MiB/s in 1s ETA: 0s
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main' locally
main: Pulling from cross-rs/aarch64-unknown-linux-gnu
56e0351b9876: Pull complete
4af1bd48d3cc: Pull complete
4aca212696d7: Pull complete
9b86d293c3aa: Pull complete
135d8bb64bfe: Pull complete
f8d31578d148: Pull complete
c31c5054dcb5: Pull complete
8b0ee46a57da: Pull complete
9ac65f18b85a: Pull complete
9c167934d71a: Pull complete
f071157538a0: Pull complete
bf17075c21f0: Pull complete
72024d4b9b48: Pull complete
670aea1b2e5b: Pull complete
3fa7105e6443: Pull complete
5d921008a306: Pull complete
fd46145107df: Pull complete
5df2ce8167df: Pull complete
Digest: sha256:69aff27a8d7e3fca1cd62d4537085f52ab6dd10a212f7fbc302193475fbe9e4f
Status: Downloaded newer image for ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main
sh: 1: cargo: not found
rustuser@rust-build:~/rustuser/hello-world$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ /usr/bin/docker
+ /usr/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup --verbose component add rust-src --toolchain 1.70.0-x86_64-unknown-linux-gnu
verbose: read metadata version: '12'
info: downloading component 'rust-src'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2023-06-01/rust-src-1.70.0.tar.xz'
verbose: downloading with reqwest
verbose: checksum passed
info: installing component 'rust-src'
verbose: creating temp directory: /usr/local/rustup/tmp/mn72qzlcdzrry82n_dir
verbose: creating temp file: /usr/local/rustup/tmp/8shhcidths2cr1wx_file
verbose: creating temp file: /usr/local/rustup/tmp/6rfg9gh32kmpz1vv_file
verbose: deleted temp directory: /usr/local/rustup/tmp/mn72qzlcdzrry82n_dir
verbose: creating temp file: /usr/local/rustup/tmp/h43iqhs92bv3mr6i_file
verbose: creating temp file: /usr/local/rustup/tmp/53revexnd2nzs38s_file
verbose: deleted temp file: /usr/local/rustup/tmp/8shhcidths2cr1wx_file
verbose: deleted temp file: /usr/local/rustup/tmp/6rfg9gh32kmpz1vv_file
verbose: deleted temp file: /usr/local/rustup/tmp/h43iqhs92bv3mr6i_file
verbose: deleted temp file: /usr/local/rustup/tmp/53revexnd2nzs38s_file
+ /usr/bin/docker inspect rust-build
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/home/rustuser/.xargo' -e 'CARGO_HOME=/usr/local/cargo' -e 'CROSS_RUST_SYSROOT=/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-
x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e CROSS_CONTAINER_IN_CONTAINER -e 'USER=rustuser' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=70' -e 'CROSS_RUSTC_PATCH_VERSION=0' --name cross-1.70.0
-x86_64-unknown-linux-gnu-4f472-90c541806-aarch64-unknown-linux-gnu-8913b-1688026898002 --rm --user 1000:1000 -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/.xargo:/home/rustuser/.xargo:z -v /var/lib/docker/overlay2/kusnn1ksi
tcoorg62e3s43xo4/merged/usr/local/cargo:/usr/local/cargo:z -v /usr/local/cargo/bin -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/ru
stuser/hello-world:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-lin
ux-gnu:z,ro -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world/target:/target:z -w /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world -t ghcr.io/cross-rs/aarch64-unkn
own-linux-gnu:main sh -c 'PATH="$PATH":"/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu/bin" cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
rustuser@rust-build:~/rustuser/hello-world$ env
HOSTNAME=rust-build
PWD=/home/rustuser/rustuser/hello-world
TZ=Europe/Berlin
HOME=/home/rustuser
CARGO_HOME=/usr/local/cargo
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm
RUSTUP_HOME=/usr/local/rustup
LESSOPEN=| /usr/bin/lesspipe %s
CROSS_CONTAINER_IN_CONTAINER=true
SHLVL=1
RUST_VERSION=1.70.0
PATH=/home/rustuser/.local/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/envIf I look at the GraphDriver output from docker inspect:
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/lcict9yuf79atqnw3wmyaf88n/diff:/var/lib/docker/overlay2/s5s3g442kwq94uk2i8dl2zhf3/diff:/var/lib/docker/overlay2/233a539d76681edec13902221be6618919d025f20b6d815f3eb7e8a8142a98fe/diff:/var/lib/docker/overlay2/912d79a6732c346fc70b67c0a28a698e236ae11fe0f390b3f2b9515969b531f9/diff:/var/lib/docker/overlay2/eb2a67b15cd69939d3dc9730271610df85f519476b78a0567d391681518db33d/diff:/var/lib/docker/overlay2/862a3c349e660a3834e24ca97b91e1af7ed52eccd7d5477d35fff3fd8a8c44aa/diff:/var/lib/docker/overlay2/e15b634681e07fca087589cbe2be164100de48183587c9311fcfa1653f2db143/diff",
"MergedDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged",
"UpperDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/diff",
"WorkDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/work"
},
"Name": "overlay2"
},And again if I look at /var/lib from the outer Docker container, there's no /var/lib/docker, so I think when it tries to pass those into the inner Docker container running cross, it can't. But perhaps I don't understand how Docker resolves those overlays.
rustuser@rust-build:~/rustuser/hello-world$ ll /var/lib
total 64
drwxr-xr-x 1 root root 4096 Jun 29 09:44 ./
drwxr-xr-x 1 root root 4096 Jun 12 02:00 ../
drwxr-xr-x 1 root root 4096 Jun 29 09:44 apt/
drwxr-xr-x 1 root root 4096 Jun 29 09:44 dpkg/
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git/
drwxr-xr-x 2 root root 4096 Apr 2 13:55 misc/
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam/
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python/
drwxr-xr-x 3 root root 4096 Jun 29 09:44 sudo/
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd/
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf/
rustuser@rust-build:~/rustuser/hello-world$If I run the cross image with -it and check what is mounted, there's essentially nothing there.
rustuser@rust-build:~/rustuser/hello-world$ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/home/rustuser/.xargo' -e 'CARGO_HOME=/usr/local/cargo' -e 'CROSS_RUST_SYSROOT=/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e CROSS_CONTAINER_IN_CONTAINER -e 'USER=rustuser' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=70' -e 'CROSS_RUSTC_PATCH_VERSION=0' --name cross-1.70.0-x86_64-unknown-linux-gnu-4f472-90c541806-aarch64-unknown-linux-gnu-8913b-1688026898002 --rm --user 1000:1000 -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/.xargo:/home/rustuser/.xargo:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/cargo:/usr/local/cargo:z -v /usr/local/cargo/bin -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:z,ro -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world/target:/target:z -w /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main sh
$ id
uid=1000 gid=1000 groups=1000
$ mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ELEJ3ASXQUUB73VF6WJEB4IKBR:/var/lib/docker/overlay2/l/72G7LQQ6RMELO4SA4LZYAMFVXF:/var/lib/docker/overlay2/l/MEWX6BY6RA6H3474OXWS2XX7QE:/var/lib/docker/overlay2/l/MW7BH3LP75LSGEMMOT3HOASW4S:/var/lib/docker/overlay2/l/24FK3GPTOYU4LEHLPSEX34JKWC:/var/lib/docker/overlay2/l/QMAOC23WTEP3XE7FA3F4EWVSHX:/var/lib/docker/overlay2/l/LII7A4KYM6SQPO65RN4W3V5VMY:/var/lib/docker/overlay2/l/KZ2GJ557I3IAETSNDOFIUHFOFB:/var/lib/docker/overlay2/l/7SI72CJ5KXA4OBK72AMKA7XO6N:/var/lib/docker/overlay2/l/U4TO6HGWKZIPEO3TZLGSRFM3RZ:/var/lib/docker/overlay2/l/524BCNZJEGUX7SNXBQDNMUPSON:/var/lib/docker/overlay2/l/A3EXFEBPJ2BUEZXPJRZQR6NP64:/var/lib/docker/overlay2/l/6K5OUG4V4S7XIDA4YPCYYAI2ZH:/var/lib/docker/overlay2/l/6UEJFQNGYQE77PDPM2FHAYGXOU:/var/lib/docker/overlay2/l/KGLRUR4I7B4PB5HZ47SSLRZZKR:/var/lib/docker/overlay2/l/K4VKZY4JNRZ7Z2RAKPO2ENJONS:/var/lib/docker/overlay2/l/E4WZ7OBV4T7SX5FHRVEON3WY5C:/var/lib/docker/overlay2/l/YW3WHISABW6HOKQW5XD3ZP2TA5:/var/lib/docker/overlay2/l/DGFMBIJIULXCTE2CPAEMJIOYIE,upperdir=/var/lib/docker/overlay2/81d85e2358468765c33f9e21a2d7a05a4c2ef13e4b59565d4cc90a8740627255/diff,workdir=/var/lib/docker/overlay2/81d85e2358468765c33f9e21a2d7a05a4c2ef13e4b59565d4cc90a8740627255/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /target type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/resolv.conf type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hostname type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hosts type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/local/cargo type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /home/rustuser/.xargo type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/local/cargo/bin type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu type ext4 (ro,relatime)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/asound type tmpfs (ro,relatime,inode64)
tmpfs on /proc/acpi type tmpfs (ro,relatime,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,inode64)
$ ls /
base-runner.sh bin boot common.sh dev etc home lib lib.sh lib32 lib64 libx32 linux-image.sh linux-runner media mnt opt proc qemu root run sbin srv sys target tmp usr var
$ ls /usr/local/cargo
bin
$ ls /usr/local/cargo/bin
$ ls /usr/local/rustup
ls: cannot access '/usr/local/rustup': No such file or directory
$ ls /usr/local/
bin cargo doc etc games include lib libexec man sbin share srcAhh, let me add the docker-compose.yml I'm using for the outer Docker container:
version: '3.5'
services:
rust-build:
build:
context: containers/rust-build
environment:
- CROSS_CONTAINER_IN_CONTAINER=true
group_add:
- docker
hostname: rust-build
image: rust-build:latest
stdin_open: true
tmpfs:
- /tmp:exec
user: rustuser
volumes:
- ${HOME}:/home/rustuser
- ${HOME}/.ssh:/home/rustuser/.ssh
- /var/run/docker.sock:/var/run/docker.sock # needed to run 'cross' within container
working_dir: /home/rustuserAnd the Dockerfile for that:
FROM rust:latest
ARG USERNAME=rustuser
ARG GROUPNAME=rustuser
ARG USER_UID=1000
ARG USER_GID=1000
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Europe/Berlin
RUN mkdir -p /usr/share/man/man1/ \
# Configure apt and install packages
&& apt-get update -y \
&& apt-get -y --no-install-recommends install \
docker.io \
less \
nano \
sudo
# Create a non-root user to use
RUN groupadd --gid $USER_GID $GROUPNAME || true \
&& useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME
# Hacky solutions to make sure user is part of docker group
#
# You have to set the inner gid to the same integer as the host system docker gid
# otherwise inner docker can't talk to /var/run/docker.sock
RUN groupmod -g 113 docker \
&& adduser $USERNAME docker
lumasepa
commented
1 year ago I'm facing the same problem, any ideas on how to solve it? If there is a solution I can work on it.
Emilgardis
commented
1 year ago I'm facing the same problem, any ideas on how to solve it? If there is a solution I can work on it.
@lumasepa are you also setting CROSS_CONTAINER_IN_CONTAINER=1 ?
lumasepa
commented
1 year ago @lumasepa are you also setting CROSS_CONTAINER_IN_CONTAINER=1 ?
I tried with and without that env var, but in my case I have dind with two docker demons, one for the outer container and the other running inside the outer container to run the inner container. So host has the crate directory, outer mounts the crate directory and has the rust tooling + cross, cross runs in the outer with its container having access to rust tooling.
I changed the setup to use outer docker and inner podman and it works without problems, and I don't have to run docker demon inside of docker.
Sorry for the late response, I've been busy with other stuff, I came back to this today.
Checklist
Describe your issue
Using a clean Ubuntu Server 22.04.2 LTS system, running
crossinside of one Docker container gives me the same errorBuild fails with sh: 1: cargo: not foundlisted in https://github.com/cross-rs/cross/issues/260.If
/var/lib/dockeris not bind mounted to the outer Docker instance, when cross goes to forward some of those mounts, there is nothing to forward to the inner Docker instance:If you run the container above and drop into a shell, you can examine the paths available in /var/lib and see what is missing.
I don't have a solution to this yet. It should be possible to bind mount
/var/lib/docker, but theroot:rootpermissions will be an issue.System Info
Host system:
Outer Docker is rust:latest (debian:11, 11.7, bullseye, bullseye-20230612):
Inner Docker is the aarch64-unknown-linux-gnu image.
What target(s) are you cross-compiling for?
aarch64-unknown-linux-gnu
Which operating system is the host (e.g computer cross is on) running?
What architecture is the host?
What container engine is cross using?
cross version
cross 0.2.5
Example
https://github.com/cross-rs/cross/issues/260#issuecomment-1611735744
Examining the Docker command line that is run by cross, it finds the
/var/lib/dockermounts, I believe by querying thedocker.sock, but because the outer Docker container does not mount/var/lib/docker, the inner Docker can't then access any of those mounts.It would be an issue anyways as the outer Docker container marks all mounts
root:rootin/var/lib/docker.Additional information / notes
I'm trying to think of workarounds.
One of which would be to run
rustupin my non-admin account in the host system, and then run theaarch64-unknown-linux-gnucontainer.Another workaround might be
Cross.tomloffering a way to specify what folders to mount for which tools.Another workaround might be to use a Docker managed volume for the tools and cross.
Final option would be to reproduce the mounts in a
docker-compose.ymland reuse the cross-rs containers from the host system.