Buck2 should be built, tested, developed, and released with... Buck2!

Like it says on the tin. I think this is already the day to day at Meta (I assume), but this should be how it works in public, too! I think this is a difficult, long-term goal — but it is, I think, a worthy one. Actually, it's not even that far off right now, you can do it — and it's actually super easy thanks to reindeer vendoring, now! I was really impressed, and the build was quick — but it isn't quite as easy as it should be, perhaps, and I think we should strive to make it the main way to build the code. Right now, as of this ticket, it's also one that is probably not worth prioritizing at the expense of many other things. Mainly? I don't think we can capitalize quite yet on a lot of the most ideal benefits (caching, RBE) until several things get polished off. Also, there are lots of open questions to answer and related quandaries about features, support, and future plans.

Some things I can think of off the top of my head that need to be fixed before we can really really buy into this even being practical:

No jemalloc or optimizations in the bootstrapped build, right now
- Both are pretty critical (optimization is mandatory, jemalloc could maybe be replaced with something simpler if we try?)
There needs to be a release schedule and some binaries we can start using to bootstrap things. Otherwise, we can't even point someone to a reasonable binary to start with. #292
There should be a publicly available build cache, at minimum.
- It should be populated by CI, at minimum. And used by subsequent CI steps, so builds are snappy as hell and fast.
- All platforms, on every commit.
- What caching technology? Where's the object storage go? Who runs the build servers? Can we use GitHub Actions to serve things? I don't know!
Most of the integration tests don't seem to be open source yet. Those need to be ported and slowly made open source, and put under BUCK files.
But there's no real test harness in the OSS version! I don't know what the status of things like the internal tpx are (I heard mumbles from Neil about potential open sourcing), or if the current buck2 test functionality is considered sufficient. But that's really important in the long run since a harness is kind of needed to understand how to rerun tests functionally.
And also, what should a CI integration for Buck2 even look like? I actually have no clue about the prior art here for something like Bazel.
- Presumably, we'll at least need to integrate it into GitHub of course with something like BXL, too
- At the absolute minimum, we should test the buck2-on-buck2 build in GitHub Actions, since it seems like we don't do so already. I can write a patch for this. #315
There's probably tons of optimizations to be done in RBE to make it faster in both caching and execution
- And more friendly to systems like BuildBuddy, or OTEL tracing in BuildBarn; like #256
Cargo is... a whole thing.
- Are we going to release buck2 on crates.io? Should we? Unfortunately there's no namespacing on crates.io so as the package set proliferates we'll keep adding things, and we'll probably want automation to release all +60 packages(!!!!!!!!!) correctly.
- I honestly feel crates.io is really a chore at this scale of so many packages, because every crate has a little overhead, but the crate is the unit of parallel compilation in practice, so... It's not like you actually get any value out of having to specify an identical version field for 60 packages with all the same metadata...
- Should cargo always be supported as a viable build method? I think it's actually nice, but only if it doesn't create an absurd number of code paths or something or make things worse off (more on that in a minute)
- Maybe Cargo can be supported, but only for building the literal binary? That would make it nicer for people with cargo but maybe things like complex integration tests for example could still require a full bootstrap build — in this ideal world, the bootstrap build would be the release build after all. So kind of like an inversion of the current situation; buck2 for serious builds and cargo if you just really want to try this build for some reason.
- Or maybe we shouldn't, and eventually we could abandon first-party cargo support entirely?[^1] I also think this should be considered and could be nice. After all, buck2 isn't meant for Rust programmers, but only Rust programmers care about cargo or would care to use it, I think. It's sort of just a necessity right now. And you still want binaries even if you release on crates.io, because again, not everyone writes Rust or uses cargo! And then you wouldn't need to toil over things like release automation, etc. Just a thought.

About the build complexity thing: how many ways can buck2 even be built right now? There's fbcode_build, buck_oss_build... external OSS cargo builds? I think also Meta might do internal Cargo builds for reasons too? Am I missing some, I don't actually know? There's kind of a lot going on here and it does have some ramifications.[^2] If we actually stopped using Cargo, it would be great if it were possible to slim down everything in the shim/ layer so we could simplify that to "OSS Buck2 build" and "Meta-Internal Buck2 build".

[^1]: I believe the generated rules from Reindeer do use cargo to compile crates, but that is of course hidden from the user. This is more about running cargo build or buck2 build when you want a fresh buck2 binary.

[^2]: I actually wanted to do a change to use CARGO_PKG_VERSION in one spot in #256 when emitting build metadata, but oops — it isn't always available in other builds like buck_oss_build, since it's not using cargo! So that's a chore, where does the version number come from? How do you sync it with cargo builds? Generally having so many build variants makes small things like this annoyingly complex.

All these basically mean that like, I would only use buck2 to bootstrap buck2 if I was quite literally involved in fixing these! Once they're fixed, everyone can be happy, but until then it's a rough world. That's how it is sometimes, but the goal, of course, is to change that!

But, aside from the benefits of dogfooding, and as a closing out thought — as a motivating thread for filing this ticket, today I ran into an issue where I had to blow away my target/ directory from Cargo. I do this all the time I feel like. I am constantly cleaning up Cargo directories all over my computer. It's just like, I might as well set NVMe space on fire, sometimes. Occasionally, there will be bad interactions with Cargo and sometimes tools like Nix and others will do silly things like copy target/ directories that are 10GB+ into the Nix store. When those are large, that's really expensive and irritating, and that happens for a couple of reasons whenever I use things like Sapling or Jujutsu to write patches (e.g. because there's no .git repo so tools think they are free to disrespect .gitignore, etc.) There's a big complex chain of interactions here of course, but it's not nice to feel like I need to recompile things to avoid issues like that!

That can all be worked around, but then I have to rebuild dependencies, and all this other stuff. For some reason, that kind of sounds like a familiar problem! I wonder if there would be a build system where I didn't have to worry about my local space being used up, and with a persistent remote cache so deleting things wasn't harmful... Hence, this issue. It would be pretty heavenly if we could just buck2 build //:buck2 and get caching for free! We should really aim high on this one I think, I think but that's just my opinion.

For me the largest blocker of building buck2 with buck2 in open source is poor ergonomics of working with configurations. We want to be able to do

buck2 build --cfg=release :buck2
buck2 build --cfg=debug :buck2

We are currently actively working on improving it, this is the first draft.

Commenting on your points, skipping those I don't know answers to.

I think this is already the day to day at Meta

Yes.

Most of the integration tests don't seem to be open source yet.

Unfortunately. Internally we have two types of tests: isolated and relying on the rest of the repo. We want to convert more tests to isolated, and open source these isolated tests.

Should cargo always be supported as a viable build method?

Yes until we get decent IDE support. Internally we use mostly VS Code and CLion. We don't have any buck2 support for CLion.

Are we going to release buck2 on crates.io?

Probably not, but there are different opinions.

And also, what should a CI integration for Buck2 even look like? complex integration tests for example could still require a full bootstrap build

Internally integration tests look like regular unit tests written in Python, executed with buck2 test.

(Most) tests add buck2 binary as a dependency, so they build buck2 with buck2 first, and execute these integration tests with freshly built buck2.

I don't know what the status of things like the internal tpx are (I heard mumbles from Neil about potential open sourcing)

It is unlikely we will open source tpx as is (it relies heavily on company infra), more likely we will improve open source test runner.

how many ways can buck2 even be built right now? There's fbcode_build, buck_oss_build... external OSS cargo builds? I think also Meta might do internal Cargo builds for reasons too?

Four:

internal with buck2 (fbcode_build)
internal with cargo
open source with buck2 (open source == with patches applied disabling internal references)
open source with cargo

Internal cargo builds are not used (and actually not even fully functional internally because of some bindings to C++ which are not available in internal cargo builds). But

we run cargo test (unit tests, not integration tests) internally
we build it anyway as a smoke test for open source builds

I am constantly cleaning up Cargo directories all over my computer.

Yep, when buck2 works, experience is better than cargo. Cleaning cargo cache when it is corrupted is not nice, happens often to me.

I still think for beginner users, who are users rather than developers of Buck2, just having a Cargo build works better. So we need to make sure that code path continues to work. That doesn't mean we shouldn't do both though.

I still think for beginner users, who are users rather than developers of Buck2, just having a Cargo build works better. So we need to make sure that code path continues to work. That doesn't mean we shouldn't do both though.

Well, ideally with a quicker release schedule (#292), and well-oiled binaries, new users would likely almost always download a binary, right? It's actually kind of expensive to compile buck2! I'm able to compile it in 5m30s on my machine, but it's a Ryzen 5600X with 64GB of RAM and I'm using mold to improve the link time pretty dramatically; I expect it would go much worse on a weaker system...

But I also realize there's another use case which might be desirable down the line, and that is being able to consume buck2-related code inside crates.io packages — while things move fast I think that's pretty reasonable and people will want to do it. For example, people could release bespoke integrations or tools that aren't quite doable otherwise.

I think an important point to note here is that the people primarily developing and people primarily using just have different code paths they want. But I think we could support both another way:

One thing Google does for some projects like gvisor is automatically provide Go-compatible builds on a separate branch that gets automatically merged every day — but the project itself is developed with Bazel. Because we already need a Cargo.lock to be generated (at minimum by Reindeer), and we have BXL, we do have a pretty complete view of the whole process and could share a stable version of locked dependencies between the two. I think this should actually be pretty doable, right? In fact you could probably automate release automation with BXL pretty cleanly too, but have Buck itself be the main build system for everyone else.

One thing Google does for some projects like gvisor is automatically provide Go-compatible builds on a separate branch that gets automatically merged every day — but the project itself is developed with Bazel. Because we already need a Cargo.lock to be generated (at minimum by Reindeer), and we have BXL, we do have a pretty complete view of the whole process and could share a stable version of locked dependencies between the two. I think this should actually be pretty doable, right? In fact you could probably automate release automation with BXL pretty cleanly too, but have Buck itself be the main build system for everyone else.

Oh, and this would also be really good for some things like Nix, too. Things that download dependencies are kind of annoying sometimes when you want to build without the network — so when using Bazel and Buck in such a sandbox, you'll need to have two steps to a build; one where you have network access and vendor the downloads, and another step where you re-use them in the sandbox. We can probably make that work with BXL, but just emitting Cargo-compatible builds and lockfiles actually sounds like it might work pretty good for most people, now that I think about it...

Yes until we get decent IDE support. Internally we use mostly VS Code and CLion.

Speaking as a person who works on the IDE tooling side of things (but independently of the buck2 team)—we'd like to get Cargo-quality parity (not just feature checklist parity!) this half inside of VS Code. This will require a rust-analyzer/buck2 companion extension that allows CRUD-style updates for targets that people are interested in developing against (we can't add every target/crate to rust-analyzer as people navigate across a monorepo, as the IDE would slow down too much).

We don't have any buck2 support for CLion.

I'd really like to add support for JetBrains products, I don't believe there's been much traction on intellij-rust's end to add support for non-Cargo build systems (https://github.com/intellij-rust/intellij-rust/issues/5594). I'm not sure if writing a buck2 plugin for the intellij platform is in the cards this half.

facebook / buck2

Buck2 should be built, tested, developed, and released with... Buck2! #313