Cross isn't working due to docker run exiting without error

[sergi0g]

Checklist

• [X] I've looked through the issues and pull requests for similar reports

Describe your issue

I wanted to compile a project for Windows on my Raspberry Pi. I ran cross build --target x86_64-pc-windows-gnu --release in my project directory. It started pulling a docker image and tried to run it when it was done pulling. Cross crashed with an exec format error. I realized I hadn't installed amd64 binfmt support on my system so I installed it (using docker run --rm --privileged tonistiigi/binfmt --install amd64). I also manually tested if I can run the image. After confirming that I can run it, I ran the cross command again. The command exited right away, without any output. I tried again a few more times with no luck. I also tried other cross commands. Reinstalling cross or even Rust didn't help.

Here is the terminal output:

❯ cross build --target x86_64-pc-windows-gnu --release
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
703.8 KiB / 703.8 KiB (100 %)  59.8 KiB/s in 13s ETA:  0s
info: latest update on 2023-11-16, rust version 1.74.0 (79e9716c9 2023-11-13)
info: downloading component 'cargo'
  8.2 MiB /   8.2 MiB (100 %)  40.6 KiB/s in  3m 48s ETA:  0s
info: downloading component 'rust-std'
 26.3 MiB /  26.3 MiB (100 %) 919.2 KiB/s in  3m  5s ETA:  0s
info: downloading component 'rustc'
 58.7 MiB /  58.7 MiB (100 %) 725.2 KiB/s in  1m  5s ETA:  0s
info: installing component 'cargo'
  8.2 MiB /   8.2 MiB (100 %)   6.7 MiB/s in  1s ETA:  0s
info: installing component 'rust-std'
 26.3 MiB /  26.3 MiB (100 %)   6.1 MiB/s in  4s ETA:  0s
info: installing component 'rustc'
 58.7 MiB /  58.7 MiB (100 %)   6.8 MiB/s in  8s ETA:  0s
  4 IO-ops /   4 IO-ops (100 %)   3 IOPS in  1s ETA:  0s

  stable-x86_64-unknown-linux-gnu installed - (error reading rustc version)

info: checking for self-update
info: downloading component 'rust-std' for 'x86_64-pc-windows-gnu'
info: installing component 'rust-std' for 'x86_64-pc-windows-gnu'
 22.0 MiB /  22.0 MiB (100 %)   6.2 MiB/s in  3s ETA:  0s

❯ cross build --target x86_64-pc-windows-gnu --release
info: downloading component 'rust-src'
info: installing component 'rust-src'

❯

How can I further debug this issue?

What target(s) are you cross-compiling for?

x86_64-pc-windows-gnu

Which operating system is the host (e.g computer cross is on) running?

• [ ] macOS
• [ ] Windows
• [X] Linux / BSD
• [ ] other OS (specify in description)

What architecture is the host?

• [ ] x86_64 / AMD64
• [ ] arm32
• [X] arm64 (including Mac M1)

What container engine is cross using?

• [X] docker
• [ ] podman
• [ ] other container engine (specify in description)

cross version

cross 0.2.5 (44011c8 2023-10-23)

Example

No response

Additional information / notes

No response

[Emilgardis]

try adding -v to see what cross is doing

Most likely you're getting limited by memory, and it's rustup that is failing

Try directly calling the rustup command you see with -v

[sergi0g]

I ran the same command but with cargo instead of cross and it seems to be building now. If I get an error I will follow your suggestion with cross.

[sergi0g]

My build failed and so I tried running the docker command from cross -v. The docker container exits with code 2 and has no logs.

[Emilgardis]

alright, sounds like you're running out of memory but let's validate that.

Instead of running cross build ..., can you run cross-util run --interactive --target x86_64-pc-windows-gnu -- bash

you should get a prompt, now try

cargo -V

should give you the version of the toolchain

and then

cargo build --target x86_64-pc-windows-gnu

should build the crate

[sergi0g]

@Emilgardis I tried running the cross-util command but it returns to my normal shell without any prompt.

[Emilgardis]

ok, can you try

docker run --rm -it ubuntu:20.04 bash

does that work?

If yes, then lets try mounting the current directory and see if we can find it.

docker run --rm -v $PWD:$PWD -it ubuntu:20.04 ls $PWD

[sergi0g]

I can confirm both commands work properly

[Emilgardis]

ok, can you post the docker command that cross-util wants to do?

cross-util run --interactive --target x86_64-pc-windows-gnu -v -- bash

what I then want you to do is remove some of the arguments, i suspect it's something that's not working well.

This is not a cross issue anymore, but we can keep trying here anyway :)

[sergi0g]

It tries to run this:

/usr/bin/docker run --userns host --platform linux/amd64 -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/home/pi/.xargo' -e 'CARGO_HOME=/home/pi/.cargo' -e 'CROSS_RUST_SYSROOT=/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=pi' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=0' -e 'CROSS_RUSTC_PATCH_VERSION=0' -v /ext/home/dev/tpscube/lib:/ext/home/dev/tpscube/lib:z --name cross-stable-x86_64-unknown-linux-gnu-0d377-79e9716c9-x86_64-pc-windows-gnu-70fa0-1700321377385 --rm --user 1000:1000 -v /ext/home/.xargo:/home/pi/.xargo:z -v /ext/home/.cargo:/home/pi/.cargo:z -v /home/pi/.cargo/bin -v /ext/home/dev/tpscube:/ext/home/dev/tpscube:z -v /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:z,ro -v /ext/home/dev/tpscube/target:/target:z -w /ext/home/dev/tpscube -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main sh -c 'PATH="$PATH":"/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin" sh -c bash'

[Emilgardis]

oh yeah, forgot that the pi isn't x86_64 hehe, should've ran the previous ubuntu commands with --platform linux/amd64

can you try

docker run --rm --userns host --platform linux/amd64 -v /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:z,ro -it ubuntu:20.04 ls /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu

and then for the actual cross image, let's try

docker run --userns host --platform linux/amd64 -v /ext/home/dev/tpscube/lib:/ext/home/dev/tpscube/lib:z --rm --user 1000:1000  -v /home/pi/.cargo/bin -v /ext/home/dev/tpscube:/ext/home/dev/tpscube:z -v /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:z,ro -v /ext/home/dev/tpscube/target:/target:z -w /ext/home/dev/tpscube -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo -V

[sergi0g]

@Emilgardis the first command works and the second fails with no output

[Emilgardis]

ok, does

docker run --userns host --platform linux/amd64 -v /ext/home/dev/tpscube/lib:/ext/home/dev/tpscube/lib:z --rm --user 1000:1000  -v /home/pi/.cargo/bin -v /ext/home/dev/tpscube:/ext/home/dev/tpscube:z -v /ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/ext/home/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:z,ro -v /ext/home/dev/tpscube/target:/target:z -w /ext/home/dev/tpscube -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main ls

work?

[sergi0g]

No it does not. I still get no output.

[Emilgardis]

ok, can you try to reduce the command a bit and see if you can figure out what is causing it to fail?

docker run --userns host --platform linux/amd64 --rm --user 1000:1000 -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main ls

[sergi0g]

It still fails, without output.

[Emilgardis]

ok, try removing some of the arguments, maybe it's the --user 1000:1000 that's causing it

[sergi0g]

Still getting an error. I can't understand.

[Emilgardis]

you're getting an error now?

I'm trying to help you but you have to try to work with me so that we can find the problem

does

docker run --platform linux/amd64 --rm -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main ls

docker run --platform linux/amd64 --rm -t -i ubuntu:20.04 ls

or

docker run --userns host --platform linux/amd64 --rm -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main ls

or

docker run --userns host --platform linux/amd64 --rm -t -i ubuntu:20.04 ls

docker run --userns host --user 1000:1000 --platform linux/amd64 --rm -t -i ubuntu:20.04 ls

work?

[sergi0g]

All commands for the ubuntu:20.04 image are working. The others aren't. It seems there is some problem with the entrypoint of the container

[Emilgardis]

I see, I haven't heard of problems with https://github.com/cross-rs/cross/blob/main/docker/windows-entry.sh but you may be right (and I was under the impression that we didn't have a entrypoint on our windows images but you're right, we do)

if it is the entry, I suspect the problem is wine with qemu. We can override the entrypoint with --entrypoint

docker run --platform linux/amd64 --entrypoint /bin/bash --rm -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main bash

and then you can run the same commands as done in the windows-entry.sh file, except don't redirect winebot &> /dev/null and maybe you'll get an error message

export HOME=/tmp/home
mkdir -p "${HOME}"

# Initialize the wine prefix (virtual windows installation)
export WINEPREFIX=/tmp/wine
mkdir -p "${WINEPREFIX}"
# FIXME: Make the wine prefix initialization faster
wineboot &> /dev/null

[sergi0g]

wineboot returns an exec format error for /usr/bin/wine.

[Emilgardis]

then it's probably qemu messing up, I'm not sure how I could help you specifically, maybe try installing a specific version

docker run --privileged --pull always --rm tonistiigi/binfmt:qemu-v8.0.4 -uninstall amd64 -install amd64

[sergi0g]

I found the problem. Wine is installed for i386 in the container. When I installed binfmt for i386 I could run wine. I will try a proper build now.

[Emilgardis]

perfect! I think we can make this experience a bit better and warn if i386 is not executable

[sergi0g]

I tried running a build and it fails again. Wineboot fails with the message wine: chdir to /tmp/.wine-0/server-30-30dbdd : No such file or directory.

[Emilgardis]

I'm not an expert with wine, but I think wine should be looking in /tmp/wine due to the WINEPREFIX we've set

https://github.com/cross-rs/cross/blob/44011c8854cb2eaac83b173cc323220ccdff18ea/docker/windows-entry.sh#L9

it could just be using /tmp/.wine-x to hold the environment

maybe wine failed to boot or crashed after some time?

[sergi0g]

It doesn't boot. I am currently starting a codespace so I can see if there is an issue there.

[sergi0g]

@Emilgardis cross works fine in the codespace. I will close this issue as I can still build with cargo. However I recommend testing cross with windows as a target in arm64 environments, so you can make your docker container work. Thank you very much for trying to help me solve the problem.

[Emilgardis]

I've made the command fallible in #1373, so you should be able to build after that and pulling the new image, however you won't be able to test using wine