auto-add binary dependencies

[vitiral]

I would like for cargo scripts to come pre-equiped with as many batteries as possible (clap, toml, json, serde, itertools, etc) through some kind of module (maybe called just extra). Instead of this module just including the world and then cargo having to compile it every time (which would take forever), I have found a clever (and slightly hacky...) way to distribute a single library via tarbal.

The basic workflow is:

• tar the compiled crate up: $~/projects/extra# tar -cvpf extra-0.0.1-rust-1.19.0.tar target/debug/deps target/debug/.fingerprint
• untar that into the new crate that will use your lib: $~/projects/myproj# tar -xf ~/projects/extra/extra-0.0.1-rust-1.19.0.tar
• add extern crate lib to src/lib.rs but you do NOT need to add anything to Cargo.toml
• build with something like cargo rustc --bin -- --extern extra=/home/garrett/notes/rust/use_star/target/debug/deps/libextra-00ea3f9aa06a9c77.rlib

This will require the creation of the extra crate and have it distribute binary releases for every target and a long-term-service release for each new rust version. I think this is doable through fairly conventional travis and appveyor builds.

This would also allow us to do a binary distribution of libraries that are difficult to build on windows, like curl. We could set up the environment on appveyor so our users wouldn't have to!

Basic design

cargo-script will:

• know a link to where the binaries are hosted (probably just github releases page)
• when a script is called it will download the binary-lib based on the current system into a local repo (i.e. ~/.cargo/binlibs)
• it will create the temporary script directory as it already does
• it wil untar the binary-lib into the created script directory, essentially precompiling the entire "extra" crate
• it will inject the extra --extern extra=... into the cargo rustc call

[vitiral]

It looks like extra is taken. I kind of like the name of world for the binary crate (kind of a joke on "import the world"... which is what the crate will try to do!).

Also, this feature will be disabled by default and beta level at first. In the future I would hope for it to be the default. It will be enabled by including a bin-extra = "0.1.0" as one of the dependencies (in which case it will be removed and included as binary) or bin-extra = false to disable it.

[DanielKeep]

cargo-script already does extra work to cache binary dependencies for you. By default, it shoves all compiled dependencies into a single directory to avoid rebuilding them where possible.

Now, that aside: I have no desire or intention to start maintaining some curated list of crates with binary distribution. Heck, the project you describe (i.e. a meta-crate of "batteries") already exists: stdx. I also have no desire to start injecting magic into cargo-script.

What's more, this really isn't cargo-script's problem. I agree that Cargo shouldn't be recompiling things incessantly when it could be caching them. I agree that being able to provide precompiled binaries for tricky-to-compile crates would be a good idea. But those are things that should be solved by Cargo and crates.io. Duct-taping extra bits of Cargo infrastructure on to a third party sub-command just seems like madness. It would be better to teach Cargo to cache dependencies, and to acquire and install binary builds if desired, so that everyone benefits.

So, I'm closing this from a combination of: cargo-script already does what I feel is an appropriate amount of work to mitigate the "constant recompilation" problem, I don't agree with turning cargo-script into its own language (Rust + magic), and oh hell no are you pushing this infrastructure stuff on me (though I'd love to see Cargo itself do this).

[vitiral]

I'm glad we all agree :smile:

These are all really good points, and I totally see where you are coming from that you don't want to support them (or maintain them!) in your application. Also, I actually didn't realize that cargo-script already caches compiled crates -- that is really cool!

I totally agree with you that this "belongs in Cargo", I just think there needs to be some communiy level effort to get an MVP working before such a thing can happen. This seemed like the perfect place, but I'm actually re-thinking my entire approach. I'm thinking something like this needs to instead happen in the build-script instead (if it were to happen at all).

Thanks for reading the proposal and for the great feedback!