search

cross-rs / cross

“Zero setup” cross compilation and “cross testing” of Rust crates
Apache License 2.0
6.21k stars 354 forks source link

Unable to use cross build --offline in docker-in-docker environment #1478

Open fjnuLuis opened 2 months ago

fjnuLuis commented 2 months ago

Checklist

Describe your issue

How To Step On This Bug

symlink_recurse() { for f in "${1}"/*; do dst=${f#"$prefix"} if [ -f "${dst}" ]; then echo "invalid: got unexpected file at ${dst}" 1>&2 exit 1 elif [ -d "${dst}" ]; then symlink_recurse "${f}" else ln -s "${f}" "${dst}" fi done }

symlink_recurse "${prefix}" '

Try To Analyze This Bug

total 0 drwxr-xr-x 4 root root 31 Apr 28 07:49 . drwxr-xr-x 4 root root 31 Apr 28 07:49 .. drwxr-xr-x 4 root root 136 Apr 28 05:34 .cargo drwxr-xr-x 2 root root 6 Apr 28 07:49 bin


- It was found that `/root/.cargo` was copied to the `/cross/cargo/.cargo` directory, causing all caches to become invalid. Since the directory was not empty, mount entered a bin directory, resulting in `home:: cargo_home_with_cwd_env (x, x)` cannot automatically switch to `/cross/cargo/.cargo`
    - [home::cargo_home_with_cwd_env](https://docs.rs/home/latest/src/home/env.rs.html#64) 
- I tried many times, but couldn't get around the problem of incorrectly setting `CARGO_HOME=/cargo`, so I started trying to change the cross source code in my own workspace

### Try To Solve This Bug
- I tried to replace this line of code, but I don't know if it will cause other problems：
    - from :https://github.com/cross-rs/cross/blob/v0.2.5/src/docker/remote.rs#L282
    - to:
```rust
        if container_path_exists(engine, container, &dst, msg_info)? {
            for entry in fs::read_dir(cargo_dir)
                .wrap_err_with(|| format!("when reading directory {:?}", cargo_dir))?
            {
                let file = entry?;
                copy_volume_files(engine, container, &file.path(), &dst, msg_info)?;
            }
        } else {
            copy_volume_files(engine, container, cargo_dir, &dst, msg_info)?;
        }

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

What architecture is the host?

What container engine is cross using?

cross version

cross 0.2.5

Example

TEST_DIR=playground TEST_DOCKER_TAG=test_offline:latest

delete cache

sudo rm -rf $TEST_DIR

init a cargo project with log dependency

mkdir $TEST_DIR && cd $TEST_DIR cargo init cargo add log

build rust compilable image from cross based image

cat > Dockerfile << EOF from ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5

install rustup & cargo

RUN curl --retry 3 https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain stable ENV PATH=/root/.cargo/bin:$PATH

install cross

RUN cargo install cross

install aarch64 target

RUN rustup target add aarch64-unknown-linux-gnu

install rust-src

RUN rustup component add rust-src

install docker

RUN apt update \ && apt install apt-transport-https ca-certificates curl -y \ && install -m 0755 -d /etc/apt/keyrings \ && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor --yes -o /etc/apt/keyrings/docker.asc \ && chmod a+r /etc/apt/keyrings/docker.asc \ && echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ tee /etc/apt/sources.list.d/docker.list > /dev/null \ && apt update \ && apt install docker-ce-cli -y EOF

docker build -t $TEST_DOCKER_TAG .

cross build offline using docker-in-docker mode

docker run \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(pwd):/workspace \ -w /workspace \ -it $TEST_DOCKER_TAG /bin/bash -c "CROSS_REMOTE_COPY_REGISTRY=1 CROSS_REMOTE=1 CROSS_REMOTE_COPY_CACHE=1 cross build --offline --target aarch64-unknown-linux-gnu --verbose"

cd -



### Additional information / notes

_No response_
Emilgardis commented 2 months ago

I'm not able to try this right now, but did you get the same result on the main branch?

cargo install cross --git https://github.com/cross-rs/cross

fjnuLuis commented 2 months ago

symlink_recurse() { for f in "${1}"/*; do dst=${f#"$prefix"} if [ -f "${dst}" ]; then echo "invalid: got unexpected file at ${dst}" 1>&2 exit 1 elif [ -d "${dst}" ]; then symlink_recurse "${f}" else ln -s "${f}" "${dst}" fi done }

symlink_recurse "${prefix}" '

Emilgardis commented 2 months ago

I think we should apply your suggested fix. However, are you sure you can't use CROSS_CONTAINER_IN_CONTAINER instead? Do you need remote?

fjnuLuis commented 2 months ago

Yes, my docker storage driver is not overlay2, and this is not a service I can control, so I can’t use CROSS_CONTAINER_IN_CONTAINER and can only copy /root/.cargo to the new docker container.