vendor = false is quick & easy to manage deps & buckify, but pretty bad day to day.
Doesn't work offline as buck2 issues HEAD requests constantly
Terrible DX annoyance with "too many open files" errors due to buck trying to download 1000 crates at once. The standard start to your day looks like "run buck2 build about 15 times until by random chance the scheduler manages to get past those errors"
Those crates get downloaded again, and again, and again
reindeer buckify takes 2 seconds or so. Pretty convenient.
[vendor] ... is slow to manage deps & buckify.
Neat for small projects
Also probably neat for Meta with y'all's funky EdenFS etc.
But middle ground is bad
Middle = vendor directory of 1000 crates, 1.2 GB, 50k source files. Mostly from dupes of the windows crates which can't be pinned to one single version etc.
reindeer vendor takes 35 seconds
reindeer buckify takes 20 seconds
git status takes 150ms
The vendor folder wrecks git performance simply by its existence.
reindeer vendor ultimately just writes a bunch of .crate files into vendor/, which are just gzipped tarballs
.crate files stored in git, but using git-lfs if you like. Suddenly windows-0.48.0.crate is just a blob. Your diffs are much simpler when you modify deps. Etc.
Some buck2 rule to extract them. (There is no prelude rule that can do this with strip_prefix and sub_targets support, but prelude's extract_archive could probably have that added.)
reindeer vendor and reindeer buckify both take 2 seconds
git status takes 20ms
Buck builds are a compromise, but a pretty great one. It still has to extract the tarballs when you want to build things. But at least buck won't ever actually extract windows-0.48.0.crate on linux, and you only pay for what you build.
The DX annoyance factor during builds is back to zero. No more too many open files errors.
DX annoyance when updating deps is acceptable.
Problems:
Relies on https://github.com/dhovart/cargo-local-registry being installed. Note however this is a single-file binary. I think if you rewrote it without the dependency on the cargo crate it would be maybe a 2-file crate. And we could use it as a library.
I think storing the local registry's index folder (nested ab/ ac/ ah/ ... folders) might be a little bit annoying if you're making concurrent edits on different branches. But you can always regenerate.
The problem is a goldilocks one.
vendor = false
is quick & easy to manage deps & buckify, but pretty bad day to day.reindeer buckify
takes 2 seconds or so. Pretty convenient.[vendor] ...
is slow to manage deps & buckify.vendor
directory of 1000 crates, 1.2 GB, 50k source files. Mostly from dupes of the windows crates which can't be pinned to one single version etc.reindeer vendor
takes 35 secondsreindeer buckify
takes 20 secondsgit status
takes 150msI think we need a solution for the middle ground:
vendor = "local-registry"
, using https://github.com/dhovart/cargo-local-registryreindeer vendor
ultimately just writes a bunch of .crate files into vendor/, which are just gzipped tarballswindows-0.48.0.crate
is just a blob. Your diffs are much simpler when you modify deps. Etc.strip_prefix
andsub_targets
support, but prelude'sextract_archive
could probably have that added.)Outcomes:
although doesn't handle cargo).package = { git = "..." }
deps yetreindeer vendor
andreindeer buckify
both take 2 secondsgit status
takes 20mswindows-0.48.0.crate
on linux, and you only pay for what you build.Problems:
cargo
crate it would be maybe a 2-file crate. And we could use it as a library.index
folder (nestedab/ ac/ ah/ ...
folders) might be a little bit annoying if you're making concurrent edits on different branches. But you can always regenerate.