I'm not 100% sure the README example works as expected?

[brandonros]

README example:

FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder 
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json
# Build application
COPY . .
RUN cargo build --release --bin app

# We do not need the Rust toolchain to run the binary!
FROM debian:bookworm-slim AS runtime
WORKDIR /app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["/usr/local/bin/app"]

It could be a mistake on my end but I have a question about this part:

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

Shouldn't this be

FROM chef AS planner
COPY Cargo.toml .
COPY Cargo.lock .
# let cargo chef mock a src/main.rs or src/lib.rs for dummy project
RUN cargo chef prepare --recipe-path recipe.json

so that the way Docker-image caching works, the recipe is only rebuilt + re-ran + re-calculated if dependencies change?

I've been using cargo-chef and unless I'm using it incorrectly or misunderstanding something, if I make a 1 whitespace change in the Rust code (not in Cargo.lock/Cargo.toml), it recompiles all of the dependencies again no matter what instead of benefitting from image layer caching?

[LukeMathWalker]

No, the example is correct. The COPY statement only invalidates the planner stage of the build. The building stage gets invalidated if and only if recipe.json changes.

[brandonros]

Is it out of scope for this tool to "cache" the compilation of the 3rd party dependencies (Cargo.toml) and not need to re-compile them every time there is a src/ code change?

[LukeMathWalker]

That's precisely what the tool does 👀 If that's not happening on a specific project of yours, let's open a bug report to see what's wrong. If this is a theoretical question, the launch blog post has a longer explanation as to why this works as expected.

[brandonros]

@LukeMathWalker sorry to bother you

Try to build just the dependencies in the planner layer, not properly stubbing the lack of a src/main.rs

FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef

# planner
FROM chef AS planner
WORKDIR /app
COPY Cargo.lock ./
COPY Cargo.toml ./
# notice the lack of src/ here
RUN cargo chef prepare --recipe-path recipe.json

# builder
FROM chef AS builder
WORKDIR /app
# hydrate
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json
# build
COPY src/ src/
COPY Cargo.lock .
COPY Cargo.toml .
RUN cargo build

# runtime
FROM debian:stable-slim AS runtime
# copy app from builder
WORKDIR /app
COPY --from=builder /app/target/debug/rust_api /app/rust_api
# entrypoint
ENTRYPOINT ["/app/rust_api"]

Error:

Untagged: rust-api:0.0.1
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 633B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest-rust-1
#2 ...

#3 [internal] load metadata for docker.io/library/debian:stable-slim
#3 DONE 0.3s

#2 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest-rust-1
#2 DONE 0.6s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [runtime 1/3] FROM docker.io/library/debian:stable-slim@sha256:f8bbfa052db81e5b8ac12e4a1d8310a85d1509d4d0d5579148059c0e8b717d4e
#5 DONE 0.0s

#6 [chef 1/1] FROM docker.io/lukemathwalker/cargo-chef:latest-rust-1@sha256:853e9beab843a8bebd6ca928432b235e0fff149e95b0a9abf663d71950968de8
#6 resolve docker.io/lukemathwalker/cargo-chef:latest-rust-1@sha256:853e9beab843a8bebd6ca928432b235e0fff149e95b0a9abf663d71950968de8 0.0s done
#6 ...

#7 [internal] load build context
#7 transferring context: 8.69kB done
#7 DONE 0.1s

#6 [chef 1/1] FROM docker.io/lukemathwalker/cargo-chef:latest-rust-1@sha256:853e9beab843a8bebd6ca928432b235e0fff149e95b0a9abf663d71950968de8
#6 sha256:61c8ff7dc9e0f997e71be81ceb92c20e9e2fa28daa81025b1d5ee507de461273 5.10kB / 5.10kB done
#6 sha256:c5e3e6923d260a3f013cd09f4aa6a3f0149d17e1219823e4bcf5fd35b4c7462c 1.45kB / 1.45kB done
#6 sha256:853e9beab843a8bebd6ca928432b235e0fff149e95b0a9abf663d71950968de8 1.61kB / 1.61kB done
#6 sha256:14a68e8af3196c8f487ae85f7a116e952dfca322c2ee2e6d957a915e0420fed8 1.05MB / 2.03MB 0.3s
#6 sha256:14a68e8af3196c8f487ae85f7a116e952dfca322c2ee2e6d957a915e0420fed8 2.03MB / 2.03MB 0.3s done
#6 extracting sha256:14a68e8af3196c8f487ae85f7a116e952dfca322c2ee2e6d957a915e0420fed8 0.1s done
#6 DONE 0.7s

#8 [builder 1/7] WORKDIR /app
#8 DONE 0.2s

#9 [planner 2/4] COPY Cargo.lock .
#9 DONE 0.1s

#10 [planner 3/4] COPY Cargo.toml .
#10 DONE 0.3s

#11 [planner 4/4] RUN cargo chef prepare --recipe-path recipe.json
#11 0.849 Error: Failed to compute recipe
#11 0.849 
#11 0.849 Caused by:
#11 0.849     0: Cannot extract Cargo metadata
#11 0.849     1: `cargo metadata` exited with an error: error: failed to parse manifest at `/app/Cargo.toml`
#11 0.849        
#11 0.849        Caused by:
#11 0.849          no targets specified in the manifest
#11 0.850          either src/lib.rs, src/main.rs, a [lib] section, or [[bin]] section must be present
#11 0.850        
#11 ERROR: process "/bin/sh -c cargo chef prepare --recipe-path recipe.json" did not complete successfully: exit code: 1
------
 > [planner 4/4] RUN cargo chef prepare --recipe-path recipe.json:
0: Cannot extract Cargo metadata
0.849     1: `cargo metadata` exited with an error: error: failed to parse manifest at `/app/Cargo.toml`
0.849        
0.849        Caused by:
0.849          no targets specified in the manifest
0.850          either src/lib.rs, src/main.rs, a [lib] section, or [[bin]] section must be present
0.850        
------
Dockerfile:8
--------------------
   6 |     COPY Cargo.lock .
   7 |     COPY Cargo.toml .
   8 | >>> RUN cargo chef prepare --recipe-path recipe.json
   9 |     
  10 |     # builder
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo chef prepare --recipe-path recipe.json" did not complete successfully: exit code: 1

Is this... expected?

The alternative is

# planner
FROM chef AS planner
WORKDIR /app
COPY Cargo.lock .
COPY Cargo.toml .
COPY src/ src/ # copy entire source directory?
RUN cargo chef prepare --recipe-path recipe.json

which will not work in my opinion (unless I am wrong). If you make a 1 byte change in src/, you are rebuilding all of your dependencies again (defeating the purpose of this library?)

I believe (for anybody else reading along) the answer is "each COPY layer is separate in Docker and you really shouldn't be explicit and just use COPY . . and a good .dockerignore)" and it was my own footgun syntax was copying things explicitly causing the issue?