search

cross-rs / cross

“Zero setup” cross compilation and “cross testing” of Rust crates
Apache License 2.0
6.79k stars 379 forks source link

Running Cross in distrobox with custom RUSTUP_HOME fails #1559

Open ngraham20 opened 2 months ago

ngraham20 commented 2 months ago

I am trying to build for x86_64-pc-windows-gnu from a Linux PC using Cross. I am in a distrobox environment (fedora toolbox 40) to contain the cross-compliation stuff separate from the rest of my dev environment.

my current default toolchain is stable-x86_64-unknown-linux-gnu

I have: CARGO_HOME=/opt/rust/cargo RUSTUP_HOME/opt/rust/rustup

the permissions on /opt/rust are my_user:my_user 755

When trying to run cross build --target x86_64-pc-windows-gnu, I get the following error: Error: statfs /opt/rust/cargo: no such file or directory

Emilgardis commented 2 months ago

Im not familiar with what a distrobox is.

Can you run the command with -v and post the output? Also, what version of cross are you using?

ngraham20 commented 2 months ago

Im not familiar with what a distrobox is.

distrobox is a containerized development platform. It's a dev container, but which I have essentially symlinked podman to the host podman client (i.e. running podman ps -a from inside a distrobox actually shows the distrobox itself among the other containers)

Can you run the command with -v and post the output? Also, what version of cross are you using?

sure thing. Speedylemon is the name of the project and the distrobox container

⬢ [Speedylemon] ❯ cross build -v --target x86_64-pc-windows-gnu
+ cargo metadata --format-version 1 --filter-platform x86_64-pc-windows-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/bin/podman
+ /usr/bin/podman run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=nathaniel' -v /home/nathaniel/Repos/speedylemon/beetlerank:/home/nathaniel/Repos/speedylemon/beetlerank -v /home/nathaniel/Repos/speedylemon/feotui:/home/nathaniel/Repos/speedylemon/feotui --rm -v /home/nathaniel/.xargo:/xargo:z -v /opt/rust/cargo:/cargo:z -v /cargo/bin -v /home/nathaniel/Repos/speedylemon:/home/nathaniel/Repos/speedylemon:z -v /opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /home/nathaniel/Repos/speedylemon/target:/target:z -w /home/nathaniel/Repos/speedylemon -i -t ghcr.io/cross-rs/x86_64-pc-windows-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build -v --target x86_64-pc-windows-gnu'
Error: statfs /opt/rust/cargo: no such file or directory
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
⬢ [Speedylemon] ❯ cross --version
cross 0.2.5
[cross] note: Falling back to `cargo` on the host.
cargo 1.81.0 (2dbb1af80 2024-08-20)

I am unsure what "host" means in this case, as cross is not installed outside the distrobox container like is shown here:

❯ cross
fish: Unknown command: cross
ngraham20 commented 2 months ago

it appears it does not specifically have to do with the target

⬢ [Speedylemon] ❯ cross build
Error: statfs /opt/rust/cargo: no such file or directory
ngraham20 commented 2 months ago

and just to be thorough:

⬢ [Speedylemon] ❯ CROSS_CONTAINER_IN_CONTAINER=true cross build
Error: 
   0: `podman inspect bazzite` failed with exit status: 125

Stderr:
   Error: no such object: "bazzite"

Stdout:
   []

The name of my container is not bazzite, that's my host PC's hostname. I'm not sure where its getting bazzite from

This is the line from podman ps that includes the development container: 730395bc5d5d registry.fedoraproject.org/fedora-toolbox:40 --verbose --name ... 2 hours ago Up 2 hours Speedylemon

Emilgardis commented 2 months ago

can you try cross-util run --target x86_64-pc-windows-gnu -i -- bash and poke around the /rust and /cargo folders, check if it's empty etc.

I suspect this is an issue of redirection in filesystem. Could you also show me usr/bin/podman info and /usr/bin/podman version

ngraham20 commented 2 months ago

The cross package that's on crates.io does not have run. I tried installing the git version as described in the readme, and it does, even though cross's version is the same. This did not fix the issue by itself though

ngraham20 commented 2 months ago

can you try cross-util run --target x86_64-pc-windows-gnu -i -- bash and poke around the /rust and /cargo folders, check if it's empty etc.

⬢ [Speedylemon] ❯ cross-util run -v --target x86_64-pc-windows-gnu -i -- bash
+ /usr/bin/podman
+ /usr/bin/podman version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ /usr/bin/podman info -f '{{ .Version.OsArch }}'
+ cargo metadata --format-version 1 --filter-platform x86_64-pc-windows-gnu
+ rustc --print sysroot
+ /usr/bin/podman
+ /usr/bin/podman version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ /usr/bin/podman info -f '{{ .Version.OsArch }}'
+ /usr/bin/podman run --userns host -e 'XARGO_HOME=/home/nathaniel/.xargo' -e 'CARGO_HOME=/opt/rust/cargo' -e 'CROSS_RUST_SYSROOT=/opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=nathaniel' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=0' -e 'CROSS_RUSTC_PATCH_VERSION=0' -v /home/nathaniel/Repos/speedylemon/beetlerank:/home/nathaniel/Repos/speedylemon/beetlerank:z -v /home/nathaniel/Repos/speedylemon/feotui:/home/nathaniel/Repos/speedylemon/feotui:z --name cross-stable-x86_64-unknown-linux-gnu-af38f-eeb90cda1-x86_64-pc-windows-gnu-5e68f-1726588603760 --rm -v /home/nathaniel/.xargo:/home/nathaniel/.xargo:z -v /opt/rust/cargo:/opt/rust/cargo:z -v /opt/rust/cargo/bin -v /home/nathaniel/Repos/speedylemon:/home/nathaniel/Repos/speedylemon:z -v /opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu:/opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu:z,ro -v /home/nathaniel/Repos/speedylemon/target:/target:z -w /home/nathaniel/Repos/speedylemon -t -i ghcr.io/cross-rs/x86_64-pc-windows-gnu:main sh -c 'PATH="$PATH":"/opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin" sh -c bash'
Error: statfs /opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu: no such file or directory

This location does have content in it.

⬢ [Speedylemon] ❯ ls /opt/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu
bin/  etc/  lib/  libexec/  share/

I suspect this is an issue of redirection in filesystem. Could you also show me usr/bin/podman info and /usr/bin/podman version

⬢ [Speedylemon] ❯ /usr/bin/podman info
host:
  arch: amd64
  buildahVersion: 1.37.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 94.63
    systemPercent: 1.38
    userPercent: 3.98
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: Holographic
    distribution: bazzite
    variant: bazzite
    version: "40"
  eventLogger: journald
  freeLocks: 1975
  hostname: bazzite
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.9.12-205.fsync.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6345400320
  memTotal: 65098477568
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 1310453760
  swapTotal: 4294963200
  uptime: 45h 32m 6.00s (Approximately 1.88 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/nathaniel/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/nathaniel/.local/share/containers/storage
  graphRootAllocated: 1998694907904
  graphRootUsed: 157631664128
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/nathaniel/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.2
  Built: 1724198400
  BuiltTime: Tue Aug 20 18:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2
⬢ [Speedylemon] ❯ /usr/bin/podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.22.6
Built:        Tue Aug 20 18:00:00 2024
OS/Arch:      linux/amd64
ngraham20 commented 2 months ago

I just tried re-installing rust into the default location instead of /opt ($HOME/.cargo and $HOME/.rustup) and the build worked fine. I have a feeling that either I needed other environment variables set to make /opt/rust work, or there's something that cross expects to be in $HOME that doesn't work with custom installation locations

Emilgardis commented 2 months ago

interesting, there shouldn't be anything like that, all paths are grabbed from rustup in path

ngraham20 commented 1 month ago

I wonder if maybe I just didn't set enough environment variables for moving rust into /opt/rust?

Here are the variables I set (fish shell):

if test -d /opt/rust
    set -gx CARGO_HOME /opt/rust/cargo
    set -gx RUSTUP_HOME /opt/rust/rustup
    set -gx XARGO_HOME /opt/rust/xargo
    source "/opt/rust/cargo/env.fish"
end