search

cargo-bins / cargo-quickinstall

pre-compiled binary packages for `cargo install`
Apache License 2.0
218 stars 11 forks source link

revive quickbuild? #233

Open Milo123459 opened 1 year ago

Milo123459 commented 1 year ago

quickbuild was like quickinstall but for rust projects. it would fetch precompiled dependencies and put them in target.

this could be made to work very similarly to how cargo-quickinstall works, but could be useful to rewrite certain parts of it to make it better. I'm happy to work on this and was wondering if anyone else was interested?

NobodyXu commented 1 year ago

quickbuild was like quickinstall but for rust projects. it would fetch precompiled dependencies and put them in target.

Is it basically project-specific installation instead of global installation? There's definitely interest in it and it could be quite useful.

cargo-binstall has a similar issue opened https://github.com/cargo-bins/cargo-binstall/issues/176

Milo123459 commented 1 year ago

i figured it was more to do with quickinstall as we would be building the packages

NobodyXu commented 1 year ago

i figured it was more to do with quickinstall as we would be building the packages

Can you please elaborate on what quickbuild is?

Milo123459 commented 1 year ago

it fetched the dependencies and instead of compiling them locally it would fetch them from prebuilt versions and they would be put in target

NobodyXu commented 1 year ago

Thanks, I see you are talking about https://github.com/cargo-quick/cargo-quick

I think best way is to ask @alsuren to add your as maintainer of that repo and un-archive it, it pre-compiles dependencies while quickinstall pre-compiles binaries.

It is indeed an interesting project that can speedup compilation.

Milo123459 commented 1 year ago

yeah, especially on big projects it would be useful. i really hope to see it happen (and i want to help!)

alsuren commented 1 year ago

@Milo123459 quickbuild was a timeboxed spike that I did over the summer between jobs. I didn't manage to get it to a point where I was happy with it before I started my new job, so I closed the project and archived the repo.

If you would like to try again, please clone the repo onto your computer and have a go. It's a massive mess, so you might be better off starting from scratch. I would also suggest timeboxing a month on it or something. I got it to a point where it could build dep trees and restore them fine, but it got fingerprint mismatches for any trees with proc macro crates in (which is basically all interesting rust projects).

I am off work next week (16th-20th) so if you want I get on a video call (or in person if you're in London) and explain what my thinking was and where I got stuck. Reply here or send me an email if you're interested, and we can set something up.

Milo123459 commented 1 year ago

oh cool - i am in london! maybe something could be figured out.

i think I'm going to try figure something out, the confusing thing is just figuring out how to get an entire deptree (which u have done) but I'm sure it is confusing but can be figured out!!

alsuren commented 1 year ago

@Milo123459 I don't have any contact details for you. Send me an email (alsuren@gmail.com) with your availability for next week, to avoid polluting this thread too much.

Alternatively I guess you can poke me via the gitter link in the repo readme?

alsuren commented 9 months ago

So... this thing came up and it looks kind-of interesting:

https://github.com/denzp/cargo-wharf

(found via https://depot.dev/blog/buildkit-in-depth that was made the front page of the orange site a while back)

It seems like a dead project (4-5 years old, and doesn't support the 2021 edition or something? - https://github.com/denzp/cargo-wharf/issues/35 ), but it is basically the same idea as cargo-quickbuild.

When I was designing quickbuild, I was thinking about my package layers exactly like docker layers. I had not looked into buildx at the time, so I didn't know that docker could do anything other than linear build steps. It turns out you can encode arbitrary graphs in the thing, and have it schedule their execution and cache management.

I have not actually tried it out yet, or read the source, so I might be getting excited about nothing.

Why am I excited about this?

They claim to be able to build trees of dependencies, cached as independent layers. This means that they have either managed to match cargo's resolver when it comes to proc-macro crates, or they've managed to sidestep it somehow.

If they solved it, we could copy their approach (or just fork their code and incorporate it into a cross-platform docker-less tool.

If they found a way to sidestep it, maybe we could use the same trick to get an MVP off the ground? If it's as simple as using a pre-2021 version of cargo then that would be a really good way to get a proof of concept off the ground and use it as a call to action to get help porting it to modern rust.

There is a risk that they don't share my goal of making a globally useful build cache, and they copy the top-level Cargo.toml and Cargo.lock all the way down the tree. If that's the case then I would be sad.

Regardless of whether I manage to use cargo-wharf's ideas to revive cargo-quickbuild, it might be possible to use cargo-wharf directly, to speed up our linux builds.

NobodyXu commented 7 months ago

Amazing, let me have a look into the project and if we could use it in quickinstall