Consider wasm instead of (in addition to?) Starlark

👋, excited to see buck2 official, looks great!

Starlark was the right choice in the past, the right choice if you are a python shop, and the right choice to stick close to bazel. I can't help but think it is skating to where the puck is rather than where it will be though. Using wasm for rule definition seems like a better choice in the long term, though admittedly I haven't thought deeply about this space in a while.

Wasm would allow companies to write rules in their language of choice and perhaps even multiple langs (which sounds like a nightmare but I've seen worse🤷‍♂️). I am starting a rust-only company and have zero desire to write rules in Starlark vs Rust for example.

To reduce complexity of the rules one could sandbox execution and limit the wasm host APIs that buck2 exposes. As a transition, the project could add a starlark -> wasm compiler, essentially treating starlark as just another language that compiles to wasm.

Internally, Buck2 was developed with high compatibility with Buck1, so that meant we had to stick to Starlark. Agreed it doesn't have to be the final choice though - for example TypeScript is probably a better choice to start from than Python - which is what the Microsoft Domino build system did.

That said, I'm not sure wasm is the right choice. While you could write the rules in wasm, how would you compile them? Would you precompile them and check them in? That doesn't sound fun. Having the build system compile them runs into circularity problems - e.g. how do you compile the Rust rules to wasm, if the Rust rules themselves require compiling.

If someone can figure out an answer to that, then the starlark -> wasm is a perfectly plausible route. It would also be possible to add a run_wasm function to Starlark, as a way of bridging the gap.

I'm not exactly sure, I'll see if I can get some time to prototype.

Speaking from experience, compiling your rules into a binary that users must then download is a massive flaw at scale, IMO, one to be avoided. It's workable at medium scales but hurts still. Buck1 effectively did this (rules are all in the binary), Shake did this (rules are all in the binary), and compiling to and distributing your rules as .wasm files is basically the same thing in spirit if different in details. It creates a lot of pain. Put another way: even if .wasm was a possibility today, right now, I would still write all my rules in Starlark.

The first problem is one of bootstrapping. How do you compile the rules in the first place, and get them to users? These rules need to be in sync with every commit in the repository, because they evolve together. All of the answers here aren't good. Ideally, you'd compile your rules with a BUCK file! After all you could use rust_binary(). But then how do you write the initial rule to compile the wasm file, which will then contain all the other rules? You'd have to write those first rules in Starlark anyway to get things rolling. The alternative is to have a 'warmup' step where you run cargo build before you can run buck build, but then you've lost one of the massive advantages of Buck, it being the only build system! And you might not think this is too bad, but Buck2 is designed for really large teams. At large scale, you're going to have thousands or hundreds of thousands of lines of rule code, and the compilation step will get expensive. And when the code is that large, expensive to compile — you would really rather prefer to use Buck, because it will solve problems Cargo doesn't, circling back to step 1!

The second problem, then, is rollout of rules to user. Rules written in (let's say) any language, when consumed by BUCK files, are more or less functions. Functions are obviously an exposed API, and therefore suffer the same problems of migration and backwards compatibility as any codebase. At large scale, you will have multiple versions of the rules in deployment. This isn't speculation, it's a guarantee. So you have Buck2 version XYZ running rules.wasm built at commit 0xDEADBEEF, and another Buck2 version ABC running rules.wasm built at commit 0xCAFEBABE. These are running on two developer machines, on two different revisions of the repository. And they both expose a rule (function) called foobar_binary. You want this rule to work on both versions of the repository. But what if you want to break the API? You have to do a controlled rollout. Therefore you need some compatibility guarantee about how the API behaves. You've now slowly re-inventing semver, except it's on the build system for your own code in your own repository. This is a huge waste of resources.

Alternatively, you can mandate there is a unique, single rules.wasm binary that goes exactly with 1 commit in the repository, 1-to-1. Now you need to get a huge amount of machinery in place to make this happen (we can't commit binary files, so source this bash script on startup of your shell, which downloads this .wasm file from S3, which is identified by its SHA-256 hash, which gets put into this directory and is .gitignore'd, but make sure the CI system caches it to avoid the 20 second download penalty at start...) Also, you have to compile it. This was a huge drag with Shake in practice even at smaller scales. Not insurmountable, but a drag. I've heard of Shake build systems with over 500+ Haskell modules. Compilation at this scale really hurts iteration times without being careful.

The lesson here is: whenever you build rules into a binary form, and distribute those to users who need to run them, you are now doing Release Management. Rules are an API. That API has consumers, which are BUCK files. Anytime you need to compile the code and distribute it, you're doing release management. This was, I am sure, a huge problem with Buck1 -- you'd basically have to spin whole new Buck1 binaries for rule changes. It's a huge pain, and I didn't even use it! I just know it from experience.

Finally, consider how Buck2 now works today. There is a tiny API exposed from the core (like, 20 functions and ~15 types) on top of the Starlark core API. All rules are written in Starlark, from the ground up, and exist as source files alongside the code and BUCK files that they are used by. No more release management, no more semver, no more binary files. You need to change a rule in a backwards incompatible way? Just change all uses of it in a single commit. Add/remove/rename parameters, whatever. Doesn't matter. Need to git checkout to a version 6 months old? It's fine: the rules exist in the repository as files, so just going back in time means you get the proper set of rules for every commit, by definition. All of these problems are gone.

There is a variant of this problem: what if the core APIs exposed by the buck2 binary change? You suffer from all those same problems of rollout, compilation, etc. But here's the thing: it's a lot easier to juggle 20 small functions, and some types, than hundreds of rules with thousands of consumers.

In my experience, this design — an absolutely minimal, language-agnostic core, and all rules interpreted side by side — might be Buck2's best design pick by a large margin. And I'm no Pythonista, Starlark lover, or even an original Bazel user, to be clear.

There are a few things in tension here:

The "compile the rules to a binary, then run the binary" step massively complicates things, notably the "compile" part. That creates a circular problem that isn't easy to fix. Interpreting the language — whatever it is — solves this handily.
Coincidentally, most version control systems totally suck at binary files. So even if you did want to commit a .wasm file to the repository, solving some of those problems like rollout — it scales poorly and hurts users with slower clones, switches, or checkouts — consider something like git rebase, when asked to rebase a single change on top of 500 newer commits, will actually check out the repo to each of those 500 states one-by-one! That's a lotta churn. So then you have to do awkward stuff like put files in S3 and download them on demand, etc. Remember: Buck is designed for massive users. It's good for small ones too! But that means solutions like "it works at small scale, except at large scale where it sucks" aren't going to go over very well.
As much as I am loathe to admit it, Starlark, AKA I-can't-believe-it's-not-Python, is really part of the appeal in aggregate. I do not say this lightly. I am an avid user of the Nix language, and loved writing build systems in Shake. But ordinary programmers who want nothing to do with your build system need to write rules, and more importantly, need to write macros, so they can automate their builds effectively. Given the option to use your "favorite XYZ build system", most programmers would instead choose to shoot their computer with a gun. They don't want to spend a ton of time doing this, and want it to get out of their way so they can actually write code in their chosen language. I learned this while using Shake: it's greatest strength, which was a good programming language (Haskell), was also it's greatest weakness by a long shot. It isn't even a "Perfect-is-enemy-of-the-good" situation; it's just a "this is in your way" situation. And it's true that when you have millions of rules — yeah, Python-lite isn't great as a choice. But it's a really nasty problem, that involves nasty code and lots of hacks and cruft. Rust or Haskell don't massively change the fact that the job to be done here is, simply speaking, a dirty one. But hey, at least the Buck2 version of Starlark has type checking!

Now that said, as point 1 indicated, a different interpreted language would be very possible, I think. And maybe welcome? We can imagine that since BUCK files are "just" a bunch of function calls that their semantics are fairly "portable." For example, imagine a beautiful, interpreted Lisp that we could write rules in, and now imagine an amazing compile_haskell.lsp written in it, and then a compile_rust.bzl written in Starlark. We could imagine both compile_haskell() (implemented in Lisp) and compile_rust() (implemented in Starlark) could be used by the same BUCK file, since it's just calling functions across the two boundaries. This would probably be a ridiculous amount of work, but it would be very cool (and I'd definitely write my rules in Lisp if it had a good macro expander, just sayin'...)

But, aside from that, I think just using wasm on its own is a bad deal for the reasons mentioned above; and using it just because you only want to use Rust for everything, including rules, is a bit cart-before-horse IMO. And Buck's multi-language design is one of its best features! I love how much easier it (and things like Nix) makes working in a polyglot codebase. If you're really only going to use Rust for everything, Cargo is a better choice up front, honestly, even if it has some real pain points. It doesn't scale as well, but if you're pure Rust, you can probably go pretty far. And there's some good news: if you're really only interested in Rust, but can compromise a little — you can just ignore buck2-prelude entirely and write your own rules for running rustc, from scratch! It would take a little time, and it's true it's slightly tricky. Or just use rust_binary in buck2-prelude and update it every once in a while? It's no longer a 100% pure Rust-only codebase. But it's a tiny amount of Starlark to build a (presumably) near-infinite amount of pure rust-only code from that point on, while getting all of Buck2's advantages (hermetic, parallel, remote, caching.) It's a codebase that you own and can evolve to meet your needs. That could be a good engineering tradeoff to make, if you ask me.

I'll also relay another story I heard from a Shake user (c.a. 2016), which I was reminded of while writing this, and sort of brings this full circle. They ran into this whole problem with compiling the build system, written in Haskell, ahead of time, and syncing it with the repo. So they did something different. Instead of taking many .hs files and compiling them into a single binary up front, they instead wrote a tool that used the Haskell compiler as a library, and Shake as a library, and would scan and pick up Shake.hs files in their repository — one Shake.hs for every project, sort of like a BUCK file — and compile them on demand with the compiler, load the object file (dlopen) into memory, and then run the rules inside, 100% transparently. In effect they embedded the Haskell compiler into the same binary as the "execution engine", and then the rules (also in Haskell) were loaded on demand. So it was like the build system had... a Haskell interpreter built in!

By moving the compilation step into the build tool, they got rid of the cumbersome compilation barrier, thus eliminating the awkward distribution step and distinction between "compiler" and "interpreter". You could also do this design with Rust and buck2! buck2 could depend on the rustc crate, and run it to compile Rust code transparently, and then build the action graph! It could compile the code to wasm transparently behind the scenes at this step. But there are problems with this, wrt your original goal:

Compilation of Rust (or Haskell) is expensive, relative to what the language is being used for. In essence, BUCK files and Starlark are only used to create the action graph (conceptually just a toposorted graph of commands to run) and declare inputs to the nodes, which are run by the underlying execution engine. The commands inside that graph are "useful work", and what you want to dominate build time in practice. As a consequence, computing the action graph efficiently (so you can quickly determine what changed) is also really important! You need to do that as fast as possible so all the users time is spent actually doing useful work. That's something buck2 does really really well right now. In contrast taking time to compile .rs or .hs files first, so they can then run and create the graph — that's a lot of time spent on "not useful work." That will have an impact on the tail latency of the build loop, and will probably be worse than buck2 today in many concrete scenarios.
- For example: you can modify a .bzl file to change some code semantically, but when buck2 evaluates it, if the action graph/inputs don't change, the build still feels nearly instantaneous. The interpreter is snappy. In contrast, modifying a .rs file will cause recompilation to a new object file, every time, even if the new code results in the same action graph/inputs. This cost can be paid if there is a change somewhere, and it results in many commands being re-run due to fresh inputs. But that's a big loss in the much more common case where nothing has changed, or only very little has changed.
- Also, with wasm, it's really meant to have native code generated from it, for good performance. Compiling to wasm so the wasm can get immediately compiled to native code also hurts. And if you're just using a wasm interpreter, you might as well just skip it, if the whole thing is abstracted anyway? It'll take less time and perform better that way.
The buck2 executable now needs to specifically embed the rustc crate, and have knowledge of compiling rust — completely eliminating the reason to use wasm in the first place: abstracting language choice from the buck2 executable! If you want to use Ocaml for writing rules? Same problem.

So I think at the end of the day you're kind of stuck here. I really think the core issue is that the buck2 build system needs to have concrete knowledge of whatever language is being used to evaluate the BUCK files and compute the action graph, and it needs to run that language directly — if you don't want to end up in a form of "DLL hell for build systems" by distributing nebulous binaries. And if it needs a specific language to be chosen — sure, you can use a compiled language and hide that step, but it's probably faster and easier to just build an efficient interpreter for some language than build or reuse a compiler, frankly. And if that's the case, it might as well be high level and easy to write, too...

Speaking from experience, compiling your rules into a binary that users must then download is a massive flaw at scale, IMO, one to be avoided.

Agreed. I wasn't suggesting this 👍.

The first problem is one of bootstrapping.

That's what I want to poke around with and prototype 🙂.

But here's the thing: it's a lot easier to juggle 20 small functions, and some types, than hundreds of rules with thousands of consumers.

Indeed. I believe those should be exposed via wasm, basically a "vm" /execution env for rules.

There are a few things in tension here:

The "compile the rules to a binary, then run the binary" step massively complicates things, notably the "compile" part. That creates a circular problem that isn't easy to fix. Interpreting the language — whatever it is — solves this handily.

Agreed, this is what I want to play around with to see if it can get reasonable from a dx and code perspective as I believe the benefits could be large.

Coincidentally, most version control systems totally suck at binary files.

Agreed!

As much as I am loathe to admit it, Starlark, AKA I-can't-believe-it's-not-Python, is really part of the appeal in aggregate.

I agree! I'm just greedy 😂...I want the benefit of buck2 and don't want to give up the rust type system. I also hate systems where you have one "working" language and then have to drop down to the "power language" to debug, increase performance, or add major features. The split is a lot of cognitive overhead. Yes, most small users won't need to do this...but all large ones will.

Now that said, as point 1 indicated, a different interpreted language would be very possible, I think. And maybe welcome?

Yeah, I thought about hacking in support for other interpreters as a prototype (a lot of the ones written in rust are getting to the point of usable), and then realized that if you just do wasm all roads will eventually lead through it...the web is too large to ignore and it will pull every language into wasm eventually IMHO.

And Buck's multi-language design is one of its best features!

Totally. I want buck2 for the other benefits, and if later I need some non-rust dep (which I don't wanna but sometimes you gotta) I won't have a large project on my hands and instead can just add a simple buck rule. Then later if I rip it out, it's just deleting the rules and files. I love optionality!

FYI, I think cargo is the weakest part of the rust ecosystem...it's a local maximum of the previous generation of tools.

Thank you for the long comment with a ton of great info! Really appreciate it and I can feel your knowledge and passion in the space 🤙. I've also been lurking and seeing your contributions here, which are really great 🍻.

FYI, I managed the team at FB that created buck1 and have worked on build systems, version control, and release management at many large companies you have heard of (using both buck and bazel) 😎. So while I very much appreciate you explaining to me from zero (truly!), you can skip most of the context about scale and how buck works in the future 🚀.

While I don't have a solution for bootstrapping a compiler for compiling rules to WebAssembly (but if I were to make a guess, it would involve Starlark at some level), I'd like to ask/echo support for WebAssembly support in Buck2. For me, I think it'd be most impactful on the BXL side of things, where I'm distributing a Rust binary that invokes a set of BXL scripts. The bootstrapping concerns that Neil and Austin brought up are therefore less salient, since the binary I'm distributing is built by itself.

(I will note that while BXL has been great to use, I've found that I'm not particularly good at managing a non-trivial BXL script, where "non-trivial" is any BXL script exceeding 50 lines. Being able to write the queries and Buck graph transformations in Rust would result in an amazing developer experience.)

I like that use case! Could you approximate it today by wrapping buck2 as a lib and writing your own "frontend" that calls your logic in rust (conceptually replacing https://github.com/facebook/buck2/tree/main/app/buck2_bxl with your own code)? Come to think of it, perhaps I could do so for my use case...

I like that usecase! Could you approximate it today by wrapping buck2 as a lib and writing your own "frontend" that calls your logic in rust (conceptually replacing https://github.com/facebook/buck2/tree/main/app/buck2_bxl with your own code)?

I'd like to, but that API isn't stable for good reason and I'd still need to connect to the Buck daemon directly. The policy of the Buck team with regard to connecting to the daemon is "please don't" for a few reasons (some pertaining to stability of the API, others due to managing the daemon state that's necessary for how Buck2 is distributed inside of fb). It's a reasonable policy, and while I think I can personally manage the complexity that the Buck2 CLI handles for me, it's not a scalable approach, and there's no reason for me to not respect their wishes.

Come to think of it, perhaps I could do so for my use case...

I'm not on the Buck team, so take this with a grain of salt: I think it's a reasonable to try it out and see if meets your needs, but don't try to rely on it—I'd treat it like using a rustc crate—subject to change at any time and break you. I'd only use it to validate your hypothesis and propose a longer-term API.

(At the moment, I think the state of BXL is "early access and subject to change". Folks from Buck have refactored my BXl scripts for me as they've made changes, but I don't think that's something they can commit to outside of the fb monorepo.)

There are three places we write rules:

In BUCK files. There, the logic should hopefully be super simple, so I see Starlark as more like TOML-on-steroids. I'd actually like to one day introduce TOML or something as an alternative to BUCK files. I think writing those in a more power language probably means you have over-complicated the BUCK files, so I don't see that as particularly useful.
In Bxl scripts. There, the logic makes a lot of sense to be specified in a better language. It's also after the build graph, so bootstrapping isn't a big deal. I could easily imagine a Bxl function run_wasm(ctx, "file.wasm") or something that hops into a wasm file and runs from there.
In user defined rules. These are the ones that have bootstrapping problems, but in some ways they already do, e.g. a lot of Java tooling is written in Java, and you need to compile some of it to compile the rest of it. If you are using Rust, perhaps you don't need to write that many user defined rules, since we already have them written. This would also be the hardest bit to replace.

I see there is a new competitor for Starlark https://github.com/jetpack-io/tyson

facebook / buck2

Consider wasm instead of (in addition to?) Starlark #171