Using the stable rust compiler for buck2

[thoughtpolice]

What would it take to allow buck2 to build with a stable rust release? A recent contribution to Nixpkgs is trying to add a buck2 package. It's a fork of my package from buck2-nix. See https://github.com/NixOS/nixpkgs/pull/232471 for that.

But specifics of that PR aside, there's something of a rather large nit to pick: buck2 currently requires a very specific nightly, but this is problematic for various reasons in the upstream Nixpkgs repository. Notably, we try to pick exactly 1 stable compiler to use for all packages[1] in an ecosystem by default, including Rust. We don't package every nightly rustc build in Nixpkgs or anything like that; you can achieve that with third party code, like I do with buck2-nix. But that code is decidedly not upstream and for good reason.[1] This is not a Nixpkgs specific phenomenon either — any application that requires nightly will require downstream package managers to figure out a solution for this if they want to ship it to users. We work around it with a gross hack of exporting RUSTC_BOOTSTRAP=1, but this isn't an ideal solution and I don't like perpetuating it unless all else fails.

I realize Meta doesn't have these restrictions for stable versus nightly and my (perhaps misinformed) impression is that using nightly features is generally considered OK; after all, you can migrate things more rapidly and effectively thanks to the monorepo, you pin versions, you have refactoring tools, etc. But this would be nice for outside consumers.

If it's mostly a matter of "We didn't think too much about it, and nightly features are convenient, but would accept patches to target stable" then that's perfectly OK and something that can be worked towards. I'd be happy to write patches to try and help that. But if there's something preventing that on a technical level it'd be great to know, at least.

---

[1] You may have heard of Nix and be thinking "But isn't a major benefit of Nix that it allows you to use multiple versions of a single package?" Yes, and there are good reasons to do so. But doing it inside the core "standard library" of Nix code that ~everything depends on is a bad idea for many reasons we've found and for consistency and ease of use, it's easiest to just say "The default version is X, everything is built with X, you can use Y if you specifically choose it on a global or case-by-case basis."

[stepancheg]

So far we managed to convert starlark-rust to work in stable. Was not easy and required some conditional compilation like this to keep it fast on nightly, and working on stable:

https://github.com/facebookexperimental/starlark-rust/blob/130b365dd4233dfd5087de6d2654cc4c2f65000f/starlark/src/values/layout/const_type_id.rs#L21-L26

It is hard to say if it 100% possible to convert buck2 to work on stable, because we use a lot of features many of which we don't really need.

I'm in favor of removing as many unstable features as possible for this reason: to see clearly which features we really need, and which are small conveniences.

As for committing to support buck2 on stable, I'm not 100% sure. Even if we make buck2 work on stable today, tomorrow we may find some feature very important. But let's start by removing unnecessary features first.

(If you or anyone plans to submit PRs to remove unstable features, please do many small PRs instead of few large PRs).

[thoughtpolice]

Yeah, none of the features that I looked via ripgrep stuck out as "absolutely mandatory", but definitely "nice to have". Submitting patches sounds reasonable, I can try chipping away at the stones here to remove things on a case-by-case basis as needed.

I don't think that committing to the stable rustc version is necessary right now, and I wouldn't necessarily encourage y'all to commit to it — but in the long run, I think it should be given a serious look and thought about, especially if wider adoption is seen as something worth encouraging/supporting/whatever by the team at Meta. This is just one of those things that, while somewhat technical in nature, is also a matter of meeting users where they're at and managing their perceptions, so there're no hard answers.

At the very least, trying to keep nightly feature use to a minimum, and possibly removing those uses, means that:

• We're less likely to trip up on changes or bugs to the compiler, as nightly advances.
  • Basically, it reduces risk. Less bugs encountered, less potential for change if features (like box syntax) are removed.
  • For example, the build.rs script for buck2-core might be less eager to reject incorrect versions if the risk to different versions is much lower! Maybe I'd accept using RUSTC_BOOTSTRAP=1 in Nixpkgs, in such a case...
• We'll better understand what's needed, and how much work is needed to keep it there.
  • For something like this there's no real way to know or understand what the impact will be, without trying and seeing how it shakes out, I think.
  • There's certainly some tension between ease of development and meeting users where they're at, like I said.

[ndmitchell]

I think there are some nightly features we are using for bad reasons, or where the alternative is not that much worse. Those can go. Think https://github.com/thoughtpolice/buck2/commit/c69f195e0e85d9de994358723bd1f7a40f0143eb.

There are some where it's just a simple function being added to Rust, and sooner or later it will become stable, and the alternative is either ugly or unsafe. Think https://github.com/thoughtpolice/buck2/commit/62d892d3ea70b7b3d82604aadd70c0613575a375. I'd say there is little value in removing 99% of these, but great value in removing 100% of them. That makes me reluctant to accept patches walking up to the line, unless we can actually cross it.

I'm really not worried about rustc changing and breaking things for us, as that has happened, but never with an unstable feature (apart from the removal of box, which actually wasn't a big deal). But it does make it easier for users.

[thoughtpolice]

FWIW: as #292 is now fixed, I've added a buck2 binary package to nixpkgs in NixOS/nixpkgs#243148. This can be easily and automatically updated and will work for any macOS/Linux user. You can just think of nixpkgs as an alternative distribution channel for the binary releases.

For now, this is an acceptable mitigation for all my purposes; Nixpkgs inclusion was one of my primary motivations; but many other consumers may remain unhappy about this. So, I'd welcome anyone else who might be held up on this to sign off their support here, so the Meta team can get a better idea of how much a problem this still is.

[udf2457]

@thoughtpolice The trouble with seeing nix as the solution is its still something many people don't use, and its well documented as having a steep learning curve, so its not exactly somethign people are going to want to invest time and effort learning just so they can use buck2.

I would therefore really appreciate an effort towards making buck2 work with stable rust.

[Conan-Kudo]

I would also be interested in having buck2 packaged in Fedora so we have a Starlark buildsystem tool in the distribution that doesn't rely on Java (and thus we cannot easily ship in Fedora). Unfortunately, like Nix, we really need buck2 to work with stable rust reliably, so I'd really like to see this resolved sooner rather than later.