LukeMathWalker / cargo-chef

A cargo-subcommand to speed up Rust Docker builds using Docker layer caching.
Apache License 2.0
1.72k stars 113 forks source link

Ignore workspace dependencies when cooking #261

Closed aminya closed 5 months ago

aminya commented 6 months ago

I am using cargo-chef on a workspace and want to avoid rebuilds of the external packages. However, Cargo-chef cook requires copying local packages in a workspace before cooking. Every time I change one of those files, cargo-chef needs to recook the whole thing.

Is there a way to only build the external dependencies?

FROM base as planner
WORKDIR /app

# Prepare the Cargo dependencies.
COPY ./Cargo.toml ./Cargo.lock ./packages/*/Cargo.toml ./.cargo/ ./rust-toolchain.toml ./
RUN cargo chef prepare --recipe-path recipe.json

FROM planner AS build
WORKDIR /app

# Build Cargo dependencies
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --profile=release_lto --recipe-path recipe.json

This doesn't work because of not copying the source files of the workspace dependencies:

> [server build 4/5] RUN cargo chef cook --release --recipe-path recipe.json:
0.373     Updating crates.io index
0.599 error: failed to get `mylib` as a dependency of package `server v0.0.1 (/app)`
0.599
0.599 Caused by:
0.599   failed to load source for dependency `mylib`
0.599
0.599 Caused by:
0.599   Unable to update /mylib
0.599
0.599 Caused by:
0.599   failed to read `/mylib/Cargo.toml`
0.599
0.599 Caused by:
0.599   No such file or directory (os error 2)
0.602 thread 'main' panicked at /usr/local/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-chef-0.1.65/src/recipe.rs:218:27:
0.602 Exited with status code: 101
0.602 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------

Those local dependencies are under packages and are linked to each other using {path = "../mylib" }

aminya commented 6 months ago

I found a workaround by creating empty packages. This fixes the issue, but it seems to negate the whole point of using cargo-chef to automate this process.

# Build the Cargo dependencies.
COPY ./Cargo.toml ./Cargo.lock ./rust-toolchain.toml ./
COPY ./.cargo/config.toml ./.cargo/config.toml
COPY ./packages/a/Cargo.toml ./packages/a/
COPY ./packages/b/Cargo.toml ./packages/b/
COPY ./packages/c/Cargo.toml ./packages/c/
COPY ./packages/d/Cargo.toml ./packages/d/
COPY ./packages/f/Cargo.toml ./packages/f/
RUN mkdir -p ./packages/a/src && touch ./packages/a/src/lib.rs && \
    mkdir -p ./packages/b/src && touch ./packages/b/src/lib.rs && \
    mkdir -p ./packages/c/src && touch ./packages/c/src/lib.rs && \
    mkdir -p ./packages/d/src && touch ./packages/d/src/lib.rs && \
    mkdir -p ./packages/f/src && touch ./packages/f/src/main.rs && \
    cargo chef prepare --recipe-path recipe.json && \
    cargo chef cook --profile=release_lto --recipe-path recipe.json --bin ${APP_NAME}

# Build the application.
RUN --mount=type=bind,source=packages,target=packages \
    --mount=type=bind,source=.cargo,target=.cargo \
    --mount=type=bind,source=Cargo.toml,target=Cargo.toml \
    --mount=type=bind,source=Cargo.lock,target=Cargo.lock \
    --mount=type=bind,source=rust-toolchain.toml,target=rust-toolchain.toml \
    cargo build --profile release_lto --bin ${APP_NAME} && \
    cp ./target/release_lto/$APP_NAME /app/${APP_NAME}
LukeMathWalker commented 6 months ago

Can you share the Dockerfile that doesn't cache? We also need a minimal reproducible example.

LukeMathWalker commented 5 months ago

Closing since we can't reproduce.