Rust: cache cargo cross

[idelvall]

Cargo cross requires docker or podman installed. Earthly doesn't support calling a FUNCTION within a WITH DOCKER statement, so there the rust library cannot be used to cache cargo cross runs as it is.

[idelvall]

Hi @joepio please TAL to the previous PR and let me know your thoughts.

You should be able to run it as:

DO rustc+CARGO --args="cross" --with_docker=" "

Also, you probably want to cache the cross images in your build cache. In this case, you can go like:

DO rustc+CARGO --args="cross" --with_docker="--load ghcr.io/cross-rs/{{TARGET}}:{{VERSION}} "

(see here info about cross base images)

[joepio]

Thanks @idelvall!

IMPORT github.com/earthly/lib/rust:nacho/with_docker AS rust

  WITH DOCKER
    DO rust+CARGO --args="cross" --$with_docker="--load ghcr.io/cross-rs/$TARGET:v0.2.5"
  END

Unfortunately, still results in only one RUN command allowed in WITH DOCKER

            _unknown *failed* | build target: build main: failed to solve: Earthfile line 29:2 apply BUILD +docker-musl: earthfile2llb for +docker-musl: Earthfile line 78:2 copy artifact: apply build +cross-build: earthfile2llb for +cross-build: /var/folders/4j/nbs48t1s7l52fvv4vfnrrymw0000gp/T/earthly-git3268833410/Earthfile line 159:4 only one RUN command allowed in WITH DOCKER
            _unknown *failed* | in      github.com/earthly/lib/rust:nacho/with_docker+CHECK_INITED (/var/folders/4j/nbs48t1s7l52fvv4vfnrrymw0000gp/T/earthly-git3268833410/Earthfile line 81:4)
            _unknown *failed* | called from github.com/earthly/lib/rust:nacho/with_docker+RUN_WITH_CACHE (/var/folders/4j/nbs48t1s7l52fvv4vfnrrymw0000gp/T/earthly-git3268833410/Earthfile line 52:4)
            _unknown *failed* | called from github.com/earthly/lib/rust:nacho/with_docker+CARGO (Earthfile line 52:4) --TMP_FOLDER= --args=cross --output=
            _unknown *failed* | called from github.com/atomicdata-dev/atomic-data-rust:develop+cross-build --TARGET=x86_64-unknown-linux-musl
            _unknown *failed* | in      github.com/atomicdata-dev/atomic-data-rust:develop+docker-musl --TARGET=x86_64-unknown-linux-musl --tags=joepmeneer/atomic-server:develop
            _unknown *failed* | in      github.com/atomicdata-dev/atomic-data-rust:develop+docker-all

[joepio]

Changes on my end: https://github.com/atomicdata-dev/atomic-server/pull/735 logs: https://cloud.earthly.dev/builds/2403bec4-8bb1-42cd-9c67-277fc5f9a2e2

[idelvall]

You need to remove the WITH DOCKER on your end, the function will create it internally

[joepio]

Thanks! Works, leads to new err.

Seems like it's calling cargo cross instead of cross.

I'm gonna try +RUN_WITH_CACHE instead!

EDIT: yes, that was it probably

[joepio]

Hmm nope...

     58  Error: async earthfile2llb for +docker-musl: Earthfile line 74:2 copy artifact: apply build +cross-build: earthfile2llb for +cross-build: /tmp/earthly-git3082292346/Earthfile line 82:4 apply ARG: value not supplied for required ARG: command
     59  in             github.com/earthly/lib/rust:nacho/with_docker+RUN_WITH_CACHE (Earthfile line 49:2) --command=
     60  called from    github.com/atomicdata-dev/atomic-server:improved-docker-rust-cache+cross-build --TARGET=armv7-unknown-linux-musleabihf
     61  in             github.com/atomicdata-dev/atomic-server:improved-docker-rust-cache+docker-musl --TARGET=armv7-unknown-linux-musleabihf --tags=joepmeneer/atomic-server:improved-docker-rust-cache
     62  View logs at https://cloud.earthly.dev/builds/d68fbb40-87b6-4271-b48f-aab3b67004b5

EDIT: my bad, fixed

[joepio]

made a bunch of changes, but now it can't find docker:

        +cross-build | Error:
        +cross-build |    0: no container engine found

        +cross-build | Suggestion: is docker or podman installed?
        +cross-build | earthly debugger | Command /bin/sh -c 'set -e; mkdir -p $CARGO_HOME; printf "Running:\n      $command\n"; eval $command' failed with exit code 1
        +cross-build | Entering interactive debugger
        +cross-build |

Still no access to docker it seems.

  DO rust+RUN_WITH_CACHE --$with_docker="--load ghcr.io/cross-rs/$TARGET:v0.2.5" --command="cross build --target $TARGET --release" --output="target/$TARGET/release/atomic-server"

Checks:

• I'm using IMPORT github.com/earthly/lib/rust:nacho/with_docker AS rust
• I'm passing the --$with_docker arg

[joepio]

Some of the runs keep going for hours, getting stuck, apparently

https://cloud.earthly.dev/ontola/builds/2dd9a510-d201-4963-a552-c6abf7f5749a

[idelvall]

Hi Joep, I cannot see those logs. Can you better point to a GH actions run I can check?

[joepio]

@idelvall how about this link?

[idelvall]

Hi @joepio, I've been working on a different approach. Let me know what do you think: https://github.com/earthly/lib/pull/39

[joepio]

Yay, seems to work! https://cloud.earthly.dev/builds/eeb7ac2a-0cca-41e9-86de-0e83751c1705

[idelvall]

Hey @joepio , i've seen in those logs that a significant number of crates are compiled by cross. Wondering if cross is designed to cache compiled artifacts, and if we can mount the Earthly caches into the cross container

[AlexMikhalev]

Cargo can fetch packages before build, for example for Atomic earhtfile the changes will be:

source:
  FROM +install
  COPY --keep-ts Cargo.toml Cargo.lock ./
  COPY --keep-ts --dir server lib cli  ./
  COPY browser+build/dist /code/browser/data-browser/dist
  DO rust+CARGO --args=fetch 

and then add offline:

build:
  FROM +source
  DO rust+CARGO --args="build --offline --release" --output="release/[^/\.]+"

My concern is that even way simpler step:

install:
  RUN apt-get update -qq
  RUN rustup component add clippy
  RUN rustup component add rustfmt
  RUN cargo install cross
  DO rust+INIT --keep_fingerprints=true

isn't cached between between build and test stages: I run +build and when I run +test +install re-runs with fetching all updates. Another avenue to speed up rust compilation inside earthly satellite can be to use cargo vendor command:

# -----------------
# Cargo Build Stage
# -----------------

FROM rust:1.39 as cargo-build

WORKDIR /usr/src/app
COPY Cargo.lock .
COPY Cargo.toml .
RUN mkdir .cargo
RUN cargo vendor > .cargo/config

COPY ./src src
RUN cargo build --release
RUN cargo install --path . --verbose

# -----------------
# Final Stage
# -----------------

FROM debian:stable-slim

COPY --from=cargo-build /usr/local/cargo/bin/my_binary /bin

CMD ["my_binary"]

and then re-mount vendored-sources and target folders across build/test. One more avenue - use panamax-rs to sync crates.io to the local instance before all builds.

[joepio]

I don't think they are running in parallel here.

docker-all:
  BUILD --platform=linux/amd64 +docker-musl --TARGET=x86_64-unknown-linux-musl
  BUILD --platform=linux/arm/v7 +docker-musl --TARGET=armv7-unknown-linux-musleabihf
  BUILD --platform=linux/arm64/v8 +docker-musl --TARGET=aarch64-unknown-linux-musl

This should probably happen in parallel, right? @idelvall

[idelvall]

I don't think they are running in parallel here.

docker-all:
  BUILD --platform=linux/amd64 +docker-musl --TARGET=x86_64-unknown-linux-musl
  BUILD --platform=linux/arm/v7 +docker-musl --TARGET=armv7-unknown-linux-musleabihf
  BUILD --platform=linux/arm64/v8 +docker-musl --TARGET=aarch64-unknown-linux-musl

This should probably happen in parallel, right? @idelvall

Yes they should. Let me see

[idelvall]

Hi @joepio, WITH DOCKER is affected by a parallelization issue. https://github.com/earthly/earthly/issues/3808 I've was able to overcome it by using VERSION 0.8 and making sure the target has not an inner FROM (as yours does)

[idelvall]

fixed by https://github.com/earthly/lib/releases/tag/3.0.0