Open nagisa opened 8 years ago
@nagisa, Why do you want it?
I do not remember exactly why, but I do remember that I ended just running rustc manually.
@posborne, @mcarton, @Devyn, You reacted with thumbs up. Why do you want it?
Sometimes you add a bunch of dependencies to your project, know it will take a while to compile next time you cargo build
, but want your computer to do that as you start coding so the next cargo build
is actually fast.
But I guess I got here searching for a cargo doc --dependencies-only
, which allows you to get the doc of your dependencies while your project does not compile because you'd need the doc to know how exactly to fix that compilation error you've had for a half hour :smile:
As described in #3615 this is useful with build
to setup a cache of all dependencies.
@gregwebs out of curiosity do you want to cache compiled dependencies or just downloaded dependencies? Caching compiled dependencies isn't implemented today (but would be with a command such as this) but downloading dependencies is available via cargo fetch
.
Generally, as with my caching use case, the dependencies change infrequently and it makes sense to cache the compilation of them.
The Haskell tool stack went through all this and they seemed to generally decided to merge things into a single command where possible. For fetch
they did end up with something kinda confusing though: build --dry-run --prefetch
. For build --dependencies-only
mentioned here they do have the same: build --only-dependencies
@gregwebs ok thanks for the info!
@alexcrichton, It looks like I should continue my work on the PR. Will Cargo's team accept it?
@KalitaAlexey I personally wouldn't be convinced just yet, but it'd be good to canvas opinions from others on @rust-lang/tools as well
@alexcrichton, Anyway I have no time right now)
I don't see much of a use case - you can just do cargo build
and ignore the output for the last crate. If you really need to do this (for efficiency) then there is API you can use.
What's the API?
Implement an Executor
. That lets you intercept every call to rustc and you can do nothing if it is the last crate.
I wasn't able to find any information about an Executor
for cargo. Do you have any links to documentation?
Docs are a little thin, but start here: https://github.com/rust-lang/cargo/blob/609371f0b4d862a94e2e3b8e4e8c2a4a2fc7e2e7/src/cargo/ops/cargo_rustc/mod.rs#L62-L64
You can look at the RLS for an example of how to use them: https://github.com/rust-lang-nursery/rls/blob/master/src/build.rs#L288
A question of Stack Overflow wanted this feature. In that case, the OP wanted to build the dependencies for a Docker layer.
A similar situation exists for the playground, where I compile all the crates once. In my case, I just put in a dummy lib.rs
/ main.rs
. All the dependencies are built, and the real code is added in the future.
@shepmaster unfortunately the proposed solution wouldn't satisfy that question because a Cargo.toml
won't parse without associated files in src
(e.g. src/lib.rs
, etc). So that question would still require "dummy files", in which case it wouldn't specifically be serviced by this change.
I ended up here because I also am thinking about the Docker case. To do a good docker build I want to:
COPY Cargo.toml Cargo.lock /mything
RUN cargo build-deps --release # creates a layer that is cached
COPY src /mything/src
RUN cargo build --release # only rebuild this when src files changes
This means the dependencies would be cached between docker builds as long as Cargo.toml
and Cargo.lock
doesn't change.
I understand src/lib.rs
src/main.rs
are needed to do a good build, but maybe build-deps
simply builds all the deps.
I came to this thread because I also wanted the docker image to be cached after building the dependencies. After later resolving this issue, I posted something explaining docker caching, and was informed that the answer was already linked in the stackoverflow post. I made this mistake, someone else made this mistake, it's time to clarify.
RUN cd / && \
cargo new playground
WORKDIR /playground # a new project has a src/main.rs file
ADD Cargo.toml /playground/Cargo.toml
RUN cargo build # DEPENDENCIES ARE BUILD and CACHED
RUN cargo build --release
RUN rm src/*.rs # delete dummy src files
# here you add your project src to the docker image
After building, changing only the source and rebuilding starts from the cached image with dependencies already built.
someone needs to relax...
Also @KarlFish what you're proposing is not actually working. If using FROM rust:1.20.0
.
cargo new playground
fails because it wants USER
env variable to be set.RUN cargo build
does not build dependencies for release, but for debug. why do you need that?Here's a better version.
FROM rust:1.20.0
WORKDIR /usr/src
# Create blank project
RUN USER=root cargo new umar
# We want dependencies cached, so copy those first.
COPY Cargo.toml Cargo.lock /usr/src/umar/
WORKDIR /usr/src/umar
# This is a dummy build to get the dependencies cached.
RUN cargo build --release
# Now copy in the rest of the sources
COPY src /usr/src/umar/src/
# This is the actual build.
RUN cargo build --release \
&& mv target/release/umar /bin \
&& rm -rf /usr/src/umar
WORKDIR /
EXPOSE 3000
CMD ["/bin/umar"]
You can always review the complete Dockerfile for the playground.
Hi!
What is the current state of the --deps-only
idea? (mainly for dockerization)
I agree that it would be really cool to have a --deps-only option so that we could cache our filesystem layers better in Docker.
I haven't tried replicating this yet, but it looks very promising. This is in glibc and not musl, by the way. My main priority is to get to a build that doesn't take 3-5 minutes ever time, not a 5 MB alpine-based image.
I ran into wanting this today.
@steveklabnik can you share more details about your case? Why would building only the dependencies be useful? Is starting with an empty lib.rs/main.rs not applicable to your situation?
As an aside, a cargo extension (cargo-prebuild
?) could probably move aside the main.rs
/ lib.rs
, replace it with an empty version, call cargo build
, then move it back. That would "solve" the problem.
I wanted to time a clean build, but of my crate only, and not its dependencies. The easiest way to do that is to cargo clean
followed by a hypothetical cargo build --deps-only
, then time a cargo build
.
Yeah, I mean, I could replace everything, and then pull it back, but that feels like working around things rather than doing what I actually want to do.
@shepmaster the replacing stuff gets very hairy when you have a workspace project with custom build.rs
in multiple subprojects. My Dockerfile is now very fragile and complicated.
Also on the workspace note, i don't even try to cache/shorten build times for the subprojects, only the main, so it's suboptimal as well. Any source change in a subproject means doing a full build without caching anything.
with custom build.rs
What impact does a build.rs have on your case? What do you expect a hypothetical cargo build --deps-only
to do in the presence of a build.rs? Do you believe that this behavior applies to all cases using a build.rs?
when you have a workspace
What do you expect a hypothetical cargo build --deps-only
to do in a workspace with two crates (A and B) where A depends on B? Is B built? Is A built?
What impact does a build.rs have on your case?
My (several) build.rs
are for invoking bindgen around c-libs. Some are simply: bindgen + include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
in the lib.rs
, and some lib.rs
add a bit more to the bindings
I guess my expectation of a cargo build --deps only
in a workspace scenario would be that, as a minimum, it would build any non project local deps of those submodules together with the non project local deps of my main lib (like libc
, which is often in my submodules).
I only mention build.rs
because the solution of doing empty projects, build, and then copying in the source becomes more messy when you also have a bespoke build with git submodules of various c sources built via a build.rs
.
Maybe a cargo build --deps-only
would do build only dependencies that are not local to the project. I.e. any dep that in a Cargo.toml
(root level or submodule) that references the central cargo repo, own repos, or git sources.
I think it would be useful to decide whether to accept or reject this as a desirable feature (even though there is no implementation yet) and clearly state that at the top of the issue (or a new issue).
I'd also like to have this feature. I'm also doing a Docker cache thing.
@shepmaster 's Dockerfile isn't working in my case (cross compiling stuff).
I am very sad this got closed, this would be great for docker builds, and way less hacky than current solutions. I feel docker builds in their own right are also a good enough reason alone
@cmac4603 this has not been closed
Oh my goodness! I'm so glad! Yup, totally misread, thanks @mathroc
I would love to see this for docker builds then, particularly local dev builds involving docker-compose. Generally for final test/prod builds, I use a musl builder from scratch, but for development, would be a very nice-to-have in the toolkit
downloading dependencies is available via cargo fetch
cargo fetch
downloads too much in general without having something like #5216 to allow the fetch to be constrained to a target.
I had a need for this feature today for pre-building a docker image cache. The trick to cargo init
a fresh project and reusing only one's Cargo.toml
almost worked, but I have a build.rs script, so I had to make a stub build.rs file too. It would be easier to have a "standard" way of downloading and building the deps that doesn't depend on the particulars of your build and handles your corner cases correctly.
but I have a build.rs script, so I had to make a stub build.rs file too
@golddranks Why did you need it though? If the reason is that you have build = "build.rs"
line in Cargo.toml, then just remove that line as it's not required nowadays and build.rs
is automatically assumed to be a build script if found.
Ah, didn't know that. Thanks.
Yet another reason to have this is that some CIs (AppVeyour in particular) have a limit on how much information is logged from the build. With cargo build --dependencies-only
one can build dependencies normally, then switch to cargo build -vv
for the source code that's being the primary target of the CI build.
I think we should probably consider the docker use-case in more detail. I have a hunch that there's more to the use-case than meets the eye.
I think that the CI use-case is a more interesting rationale for this feature, but I wonder if it argues instead for more control over the verbosity options (--crate-verbosity "my-crate=vv"
or something along those lines), which I could imagine being useful for other purposes ("I really want to look more closely at what's going on when I build clap
")
I came here looking for the same use-case of wanting to properly layer my Docker images to decrease build times.
I ended up with the following solution, in case this helps anyone else:
FROM ekidd/rust-musl-builder AS builder
RUN sudo chown -R rust:rust /home/rust
RUN mkdir src && touch src/lib.rs
COPY Cargo.lock .
COPY Cargo.toml .
RUN cargo build --release
ADD . .
RUN cargo build --release
FROM scratch
COPY --from=builder \
/home/rust/src/target/x86_64-unknown-linux-musl/release/my-app \
/my-app
ENTRYPOINT ["/my-app"]
Here's the result of the first run, vs a second run with only source code changes, no dependency changes:
Same docker use case here. We can do the above, but anyone reading it would wonder what's going on. It would be much clearer to have an -dependencies-only flag.
Is the question whether there are sufficient use cases (Docker, performance timing, any others?)? (I'm assuming the cargo implementation would be fairly simple.).
To be fair docker is a pretty big use case...
Same docker use case here. I would love this feature.
Same docker use case here. I don't want to have to pull and compile all the dependencies in my image everytime I change the source code.
cargo team notes:
There should be an option to only build dependencies.