cargo build --dependencies-only

[nagisa]

cargo team notes:

• Summary

---

There should be an option to only build dependencies.

[KalitaAlexey]

@nagisa, Why do you want it?

[nagisa]

I do not remember exactly why, but I do remember that I ended just running rustc manually.

[KalitaAlexey]

@posborne, @mcarton, @Devyn, You reacted with thumbs up. Why do you want it?

[mcarton]

Sometimes you add a bunch of dependencies to your project, know it will take a while to compile next time you cargo build, but want your computer to do that as you start coding so the next cargo build is actually fast. But I guess I got here searching for a cargo doc --dependencies-only, which allows you to get the doc of your dependencies while your project does not compile because you'd need the doc to know how exactly to fix that compilation error you've had for a half hour :smile:

[gregwebs]

As described in #3615 this is useful with build to setup a cache of all dependencies.

[alexcrichton]

@gregwebs out of curiosity do you want to cache compiled dependencies or just downloaded dependencies? Caching compiled dependencies isn't implemented today (but would be with a command such as this) but downloading dependencies is available via cargo fetch.

[gregwebs]

Generally, as with my caching use case, the dependencies change infrequently and it makes sense to cache the compilation of them.

The Haskell tool stack went through all this and they seemed to generally decided to merge things into a single command where possible. For fetch they did end up with something kinda confusing though: build --dry-run --prefetch. For build --dependencies-only mentioned here they do have the same: build --only-dependencies

[alexcrichton]

@gregwebs ok thanks for the info!

[KalitaAlexey]

@alexcrichton, It looks like I should continue my work on the PR. Will Cargo's team accept it?

[alexcrichton]

@KalitaAlexey I personally wouldn't be convinced just yet, but it'd be good to canvas opinions from others on @rust-lang/tools as well

[KalitaAlexey]

@alexcrichton, Anyway I have no time right now)

[nrc]

I don't see much of a use case - you can just do cargo build and ignore the output for the last crate. If you really need to do this (for efficiency) then there is API you can use.

[gregwebs]

What's the API?

[nrc]

Implement an Executor. That lets you intercept every call to rustc and you can do nothing if it is the last crate.

[gregwebs]

I wasn't able to find any information about an Executor for cargo. Do you have any links to documentation?

[nrc]

Docs are a little thin, but start here: https://github.com/rust-lang/cargo/blob/609371f0b4d862a94e2e3b8e4e8c2a4a2fc7e2e7/src/cargo/ops/cargo_rustc/mod.rs#L62-L64

You can look at the RLS for an example of how to use them: https://github.com/rust-lang-nursery/rls/blob/master/src/build.rs#L288

[shepmaster]

A question of Stack Overflow wanted this feature. In that case, the OP wanted to build the dependencies for a Docker layer.

A similar situation exists for the playground, where I compile all the crates once. In my case, I just put in a dummy lib.rs / main.rs. All the dependencies are built, and the real code is added in the future.

[alexcrichton]

@shepmaster unfortunately the proposed solution wouldn't satisfy that question because a Cargo.toml won't parse without associated files in src (e.g. src/lib.rs, etc). So that question would still require "dummy files", in which case it wouldn't specifically be serviced by this change.

[lolgesten]

I ended up here because I also am thinking about the Docker case. To do a good docker build I want to:

COPY Cargo.toml Cargo.lock /mything

RUN cargo build-deps --release  # creates a layer that is cached

COPY src /mything/src

RUN cargo build --release       # only rebuild this when src files changes

This means the dependencies would be cached between docker builds as long as Cargo.toml and Cargo.lock doesn't change.

I understand src/lib.rs src/main.rs are needed to do a good build, but maybe build-deps simply builds all the deps.

[ghost]

The dockerfile template in shepmaster's linked stackoverflow post above SOLVES this problem

I came to this thread because I also wanted the docker image to be cached after building the dependencies. After later resolving this issue, I posted something explaining docker caching, and was informed that the answer was already linked in the stackoverflow post. I made this mistake, someone else made this mistake, it's time to clarify.

RUN cd / && \
    cargo new playground
WORKDIR /playground                      # a new project has a src/main.rs file

ADD Cargo.toml /playground/Cargo.toml 
RUN cargo build                          # DEPENDENCIES ARE BUILD and CACHED
RUN cargo build --release
RUN rm src/*.rs                          # delete dummy src files

# here you add your project src to the docker image

After building, changing only the source and rebuilding starts from the cached image with dependencies already built.

[lolgesten]

someone needs to relax...

[lolgesten]

Also @KarlFish what you're proposing is not actually working. If using FROM rust:1.20.0.

1. cargo new playground fails because it wants USER env variable to be set.
2. RUN cargo build does not build dependencies for release, but for debug. why do you need that?

[lolgesten]

Here's a better version.

FROM rust:1.20.0

WORKDIR /usr/src

# Create blank project
RUN USER=root cargo new umar

# We want dependencies cached, so copy those first.
COPY Cargo.toml Cargo.lock /usr/src/umar/

WORKDIR /usr/src/umar

# This is a dummy build to get the dependencies cached.
RUN cargo build --release

# Now copy in the rest of the sources
COPY src /usr/src/umar/src/

# This is the actual build.
RUN cargo build --release \
    && mv target/release/umar /bin \
    && rm -rf /usr/src/umar

WORKDIR /

EXPOSE 3000

CMD ["/bin/umar"]

[shepmaster]

You can always review the complete Dockerfile for the playground.

[maelvls]

Hi! What is the current state of the --deps-only idea? (mainly for dockerization)

[AdrienneCohea]

I agree that it would be really cool to have a --deps-only option so that we could cache our filesystem layers better in Docker.

I haven't tried replicating this yet, but it looks very promising. This is in glibc and not musl, by the way. My main priority is to get to a build that doesn't take 3-5 minutes ever time, not a 5 MB alpine-based image.

[steveklabnik]

I ran into wanting this today.

[shepmaster]

@steveklabnik can you share more details about your case? Why would building only the dependencies be useful? Is starting with an empty lib.rs/main.rs not applicable to your situation?

[shepmaster]

As an aside, a cargo extension (cargo-prebuild?) could probably move aside the main.rs / lib.rs, replace it with an empty version, call cargo build, then move it back. That would "solve" the problem.

[steveklabnik]

I wanted to time a clean build, but of my crate only, and not its dependencies. The easiest way to do that is to cargo clean followed by a hypothetical cargo build --deps-only, then time a cargo build.

Yeah, I mean, I could replace everything, and then pull it back, but that feels like working around things rather than doing what I actually want to do.

[lolgesten]

@shepmaster the replacing stuff gets very hairy when you have a workspace project with custom build.rs in multiple subprojects. My Dockerfile is now very fragile and complicated.

[lolgesten]

Also on the workspace note, i don't even try to cache/shorten build times for the subprojects, only the main, so it's suboptimal as well. Any source change in a subproject means doing a full build without caching anything.

[shepmaster]

with custom build.rs

What impact does a build.rs have on your case? What do you expect a hypothetical cargo build --deps-only to do in the presence of a build.rs? Do you believe that this behavior applies to all cases using a build.rs?

when you have a workspace

What do you expect a hypothetical cargo build --deps-only to do in a workspace with two crates (A and B) where A depends on B? Is B built? Is A built?

[lolgesten]

What impact does a build.rs have on your case?

My (several) build.rs are for invoking bindgen around c-libs. Some are simply: bindgen + include!(concat!(env!("OUT_DIR"), "/bindings.rs")); in the lib.rs, and some lib.rs add a bit more to the bindings

I guess my expectation of a cargo build --deps only in a workspace scenario would be that, as a minimum, it would build any non project local deps of those submodules together with the non project local deps of my main lib (like libc, which is often in my submodules).

I only mention build.rs because the solution of doing empty projects, build, and then copying in the source becomes more messy when you also have a bespoke build with git submodules of various c sources built via a build.rs.

Maybe a cargo build --deps-only would do build only dependencies that are not local to the project. I.e. any dep that in a Cargo.toml (root level or submodule) that references the central cargo repo, own repos, or git sources.

[gregwebs]

I think it would be useful to decide whether to accept or reject this as a desirable feature (even though there is no implementation yet) and clearly state that at the top of the issue (or a new issue).

[chris13524]

I'd also like to have this feature. I'm also doing a Docker cache thing.

@shepmaster 's Dockerfile isn't working in my case (cross compiling stuff).

[cmac4603]

I am very sad this got closed, this would be great for docker builds, and way less hacky than current solutions. I feel docker builds in their own right are also a good enough reason alone

[mathroc]

@cmac4603 this has not been closed

[cmac4603]

Oh my goodness! I'm so glad! Yup, totally misread, thanks @mathroc

I would love to see this for docker builds then, particularly local dev builds involving docker-compose. Generally for final test/prod builds, I use a musl builder from scratch, but for development, would be a very nice-to-have in the toolkit

[axos88]

Related: https://github.com/rust-lang/cargo/issues/5198

[briansmith]

downloading dependencies is available via cargo fetch

cargo fetch downloads too much in general without having something like #5216 to allow the fetch to be constrained to a target.

[golddranks]

I had a need for this feature today for pre-building a docker image cache. The trick to cargo init a fresh project and reusing only one's Cargo.toml almost worked, but I have a build.rs script, so I had to make a stub build.rs file too. It would be easier to have a "standard" way of downloading and building the deps that doesn't depend on the particulars of your build and handles your corner cases correctly.

[RReverser]

but I have a build.rs script, so I had to make a stub build.rs file too

@golddranks Why did you need it though? If the reason is that you have build = "build.rs" line in Cargo.toml, then just remove that line as it's not required nowadays and build.rs is automatically assumed to be a build script if found.

[golddranks]

Ah, didn't know that. Thanks.

[ArtemGr]

Yet another reason to have this is that some CIs (AppVeyour in particular) have a limit on how much information is logged from the build. With cargo build --dependencies-only one can build dependencies normally, then switch to cargo build -vv for the source code that's being the primary target of the CI build.

[wycats]

I think we should probably consider the docker use-case in more detail. I have a hunch that there's more to the use-case than meets the eye.

I think that the CI use-case is a more interesting rationale for this feature, but I wonder if it argues instead for more control over the verbosity options (--crate-verbosity "my-crate=vv" or something along those lines), which I could imagine being useful for other purposes ("I really want to look more closely at what's going on when I build clap")

[JeanMertz]

I came here looking for the same use-case of wanting to properly layer my Docker images to decrease build times.

I ended up with the following solution, in case this helps anyone else:

FROM ekidd/rust-musl-builder AS builder

RUN sudo chown -R rust:rust /home/rust

RUN mkdir src && touch src/lib.rs
COPY Cargo.lock .
COPY Cargo.toml .
RUN cargo build --release

ADD . .
RUN cargo build --release

FROM scratch
COPY --from=builder \
  /home/rust/src/target/x86_64-unknown-linux-musl/release/my-app \
  /my-app
ENTRYPOINT ["/my-app"]

Here's the result of the first run, vs a second run with only source code changes, no dependency changes:

collapsed wall of text

``` ❯ time docker build -t hello .; and docker run hello Sending build context to Docker daemon 225.3kB Step 1/10 : FROM ekidd/rust-musl-builder AS builder ---> 792e654be291 Step 2/10 : RUN mkdir src/ && touch src/lib.rs ---> Running in 2208ab6053f4 Removing intermediate container 2208ab6053f4 ---> 572926d2738a Step 3/10 : COPY Cargo.lock . ---> e8eb753e4881 Step 4/10 : COPY Cargo.toml . ---> 43f3d04fe0e8 Step 5/10 : RUN cargo build --release ---> Running in 374ba4a8a864 Updating registry `https://github.com/rust-lang/crates.io-index` Finished release [optimized] target(s) in 43.65s Removing intermediate container 374ba4a8a864 ---> 843e36ec3ce7 Step 6/10 : ADD . . ---> c66cc9b7f4dc Step 7/10 : RUN cargo build --release ---> Running in bcd95a11ff47 Compiling hello-world v0.1.0 (file:///home/rust/src) Finished release [optimized] target(s) in 0.40s Removing intermediate container bcd95a11ff47 ---> 1789bdcc6ed3 Step 8/10 : FROM scratch ---> Step 9/10 : COPY --from=builder /home/rust/src/target/x86_64-unknown-linux-musl/release/hello-world /hello-world ---> Using cache ---> e8501d0c8738 Step 10/10 : ENTRYPOINT ["/hello-world"] ---> Using cache ---> fc179245d85b Successfully built fc179245d85b Successfully tagged hello:latest 53.53 real 2.17 user 0.80 sys hello world ``` ``` ❯ time docker build -t hello .; and docker run hello Sending build context to Docker daemon 225.3kB Step 1/10 : FROM ekidd/rust-musl-builder AS builder ---> 792e654be291 Step 2/10 : RUN mkdir src/ && touch src/lib.rs ---> Using cache ---> 572926d2738a Step 3/10 : COPY Cargo.lock . ---> Using cache ---> e8eb753e4881 Step 4/10 : COPY Cargo.toml . ---> Using cache ---> 43f3d04fe0e8 Step 5/10 : RUN cargo build --release ---> Using cache ---> 843e36ec3ce7 Step 6/10 : ADD . . ---> a9698b8f6a88 Step 7/10 : RUN cargo build --release ---> Running in 980a6d70a441 Compiling hello-world v0.1.0 (file:///home/rust/src) Finished release [optimized] target(s) in 0.35s Removing intermediate container 980a6d70a441 ---> 1661e048b501 Step 8/10 : FROM scratch ---> Step 9/10 : COPY --from=builder /home/rust/src/target/x86_64-unknown-linux-musl/release/hello-world /hello-world ---> d8b93e6a4ff9 Step 10/10 : ENTRYPOINT ["/hello-world"] ---> Running in e26f084ed361 Removing intermediate container e26f084ed361 ---> 6684afc6b641 Successfully built 6684afc6b641 Successfully tagged hello:latest 7.43 real 2.26 user 1.00 sys hello universe! ```

[gilescope]

Same docker use case here. We can do the above, but anyone reading it would wonder what's going on. It would be much clearer to have an -dependencies-only flag.

Is the question whether there are sufficient use cases (Docker, performance timing, any others?)? (I'm assuming the cargo implementation would be fairly simple.).

To be fair docker is a pretty big use case...

[emilk]

Same docker use case here. I would love this feature.

[anuraags]

Same docker use case here. I don't want to have to pull and compile all the dependencies in my image everytime I change the source code.