submodules

[dezren39]

it may be helpful to have submodules

nix-prelude -- current contents of buck/prelude nix-buck2 -- current contents of buck/nix nix-dotfiles -- uh.. all the dotfiles in root

and then, ideal state, a script you can curl which would inject these submodules, symlink the dotfiles, maybe rest of automated setup with nix/direnv whatever that is in setup.sh today.

so that these can be imported into others repos as submodules, like the buck2-prelude

i uh... am very beginner at nix, so i dont know that i really want to customize much of anything?

i think i could run the nix run command and then go use that as my starter template, but if i already have a repo, or but also i dont want to maintain fork and merge in upstream changes so that i can have my own src folder.

but i also want any awesome updates you make. ;-)

for now i just like, copied major portions and am winging it (badly), and am planning checking the diff occasionally, but generally i dont know others would want that, they may prefer to just occasionally pull updates for their submodule. i could have also just submoduled this repo just somewhere and then symlinked directories needed out of it, but w/e. pretend i did that.

---

i suppose if buck/nix is a submodule then nix run <...> github: might break? it would work fine if you clone it of course, but maybe that would be a ux issue if downstream consumers want to provide a nix run command to their users, like in this repo readme.

[dezren39]

maybe there is some way to use flakes to handle this outside of git? like maybe i could reference this flake in my own flake in my repo and then that would magically build everything? and then an hello-world example of this could be used as a template with nix flake init --template github:thoughtpolice/buck2-nix-template" or something? and then an update script would update the input flake and the internals wouldn't be part of the downstream repo directly

[thoughtpolice]

I'll get this part out of the way first: I'm glad you're interested in this project! It's fine if you're new to Nix, I'm still exploring stuff. Feel free to submit issues.

But this one is a pet peeve of mine: I will never, ever, ever use git submodules if given the choice. They are horrendously designed and make every change vastly more difficult for the person doing work on the repository. It would literally slow me down a ton and create way more errors in the repository. They are only slightly more convenient for downstream consumers, but most of the same issues still apply. I'll cut myself off here because if given the chance I could go on and on about submodules for a while; but it's probably better to let the scars age than show them off...

In general, this system is geared closer towards the monorepo concept than individual repos. More on that in a sec, but...

In this case, there is no real room to separate the prelude and Nix components. If you have a submodule, you need a well defined ABI between them; submodules for coupled components is a disaster. It's currently very important that they go together and spending time designing an API to decouple them is a massive up front investment without clear gains while I'm still exploring the design space about how this repository should work out. There are many possible design points to explore still and I'm thinking over several of them. You should expect this repository to change a lot. (Upshot: reading the diffs and changes might help you better understand Nix and why I did certain things! Osmosis is a good way to learn...)

In the long run, I would like this to be something you could copy/paste and just start using, using a flake template or something. Ideally, you should be able to just copy buck/, .buckconfig and mostly get it working from there (even if you don't have direnv to do some things like setup watchman.) But right now it's a bit too early for that since I'm still in design-mode.

---

There are two options for slicing up this repository that I can see in the near future, though, which I'm willing to entertain or explore, as a method for people to consume something downstream:

• josh proxy. This is basically how Git submodules should have always worked but anyway, it exists now. You can put this server in front of your git clone, and then do bi-directional (push/pull) operations on subsets of the repository. It's like git-filter-branch or git-filter-repo on steroids. With Josh, it's possible to checkout/clone a "workspace" in the repository that is actually a subset of the entire tree, and use that. You can make commits in that repo, and then still pull down changes from upstream. You can keep pull'ing as long as you want.
• EdenFS and Mononoke from Sapling, by Facebook. Sapling is a version control system. Today, you can use it with Git only. I exclusively use it for developing this repository. But Git is just one option; they're working on getting their internal source code server (Mononoke) up and running, and with that, their virtual filesystem, EdenFS, will work too. Eden allows you to clone subsets of the repository and check them out virtually. Buck2 automatically knows how to talk with EdenFS. This option is my inevitable long-term plan.

It's worth noting that both solutions, Josh and EdenFS, sort of work on the idea that upstream and downstream are connected. It's designed so that you may inevitably "go back upstream" but not a requirement. It's not a "template repository" in either case.

In the long run I have a project which I believe buck2 will be a wonderful choice for, but it will take much time, effort, and exploration to get there. I haven't decided yet if, by that point, this project will be maintained separately as a "You can use this too, if you want", or deprecated and I will just say "This repository was an experiment, it panned out, and it's now part of project Foobar. Use at your own risk and keep developing it." Time will tell, but I expect that decision is probably like, a year out at least?

I think that's a short summary. In the meantime, please feel free to keep submitting issues or questions!

[dezren39]

Yeah that second option sounds great. I don't particularly like submodules either mostly just pattern matching the buck2 repo setup. My plan is for a monorepo as well I just wasn't sure how best to keep up to date for these components within it.

I had heard something about Facebooks version control going public but didn't really know the advantages it offers. I've hardly investigated sapling either! I think I'll look into these and wait to see whether they open source that.

In the mean time I've copied from where you were a week ago and am just gonna check in on the diff occasionally. 😅

[thoughtpolice]

Yeah, right now you're probably just best off with checking in once a week/day/whatever, looking at the diffs, updating, and submitting an issue if you have Q's. Feel free!

[Fuuzetsu]

I guess the easiest way I can imagine something relatively stand-alone being useful for others is:

• make a repo and flake for buck2: this just makes buck available. This repo can re-use it if it has no patches, or not. Or use it and override. Whatever works. Mostly just the nix expression inside is useful.
• have a repo with just the nix rules, that is the prelude-ish from this repo. It's of course unclear how to keep this up to date, hence this ticket. One idea might be to simply pull it in via flake.nix and keep it up to date with nix flake lock --update-input. I'm unsure if buck is able to refer to rules outside of the repository or there needs to be an extra step to keep the nix store and current repository in sync... No idea what the prior art is: I think in bazel you are able to reference a remote repository from WORKSPACE file and you're then able to use the rules directly from there, without any fuss of having to clone/copy inside your tree. I guess if buck has something similar then that Just Works.
• have a repo (like this one!) which provides whatever extra helpers one might want: setup scripts, direnv, …? I think for most people who have their own project, they only really care about the buck2 flake rules prelude, maybe direnv stuff. Everything else is probably highly optional.

Just to be clear, I'm not saying this is what this repo should become! But I think it's valuable if it happens. I may investigate splitting off buck2 expression into a flake that can be pulled in (and overriden with different source version etc) and I'll have to look into if buck can load rules from somewhere that's not just bunch of files in working tree.

[thoughtpolice]

Regarding point 1, honestly, I think we should probably just try to get buck2 directly into Nixpkgs upstream. I have an idea about how to do this, but I'll have to draft a PR and some explanation to the upstream buck2 repo... Also, we should just add a buck2 package to the upstream buck2 flake — it was added a few weeks ago, by me, but right now it's only used for an interactive development environment via direnv so you can compile buck2 itself. It should also provide a package that downstream flake users can use.

[Fuuzetsu]

Just as a brief follow-up, I found the below stand-alone repo: https://github.com/facebook/buck2-prelude

They explicitly mention that usually one would use a submodule...

EDIT: the main guide also says that's what happens https://buck2.build/docs/getting_started/#creating-your-first-hello_world-project

---

Regarding point 1, honestly, I think we should probably just try to get buck2 directly into Nixpkgs upstream.

Right. I assumed this wasn't done because buck2 is changing fast and updating nixpkgs all the time is annoying: both for the person doing the updated but also for all the users. Often using same nixpkgs and just updating the piece of software works out quite well, say by passing own nixpkgs as input to the buck2 flake.

[thoughtpolice]

Yeah, buck2-prelude is the recommended path for most users. It's also the same code Meta uses, so it's proven. See also issue #1, which talks about the design of this repository and its custom rules vs buck2-prelude. (That said I disagree about the use of a submodule still and I'm not alone in that, but it isn't really relevant here.) Frankly you might be better off using buck2-prelude; I have my own reasons for designing this like I said but one was to educate myself. It may not pan out.

As for the nixpkgs stuff: One reason why there's no stable release yet is mostly because the Meta team doesn't need one, they just use tip-of-trunk directly and keep updating. Part of why that's possible is one of Buck2's biggest advantages: the rules are completely "userspace" and written in Starlark, and committed in the repository, and there is no language or tool specific code in the buck2 binary. Therefore the prelude/starlark code you use and the binary itself have very little "friction" against each other and are highly decoupled. The surface area API of the buck2 executable is very small and so things like changes are much easier to rollout and organize. Therefore I don't think it would be difficult to include buck2 in nixpkgs; it's already very stable. It's mostly a matter of stable releases being tagged on a regular cadence, which is what I'd propose to the upstream team, and then tagging those releases and bumping packages.

I think having the flake and having the upstream package are ultimately both necessary; flake consumers can use the flake (thus allowing them to pin whatever version they want and match with whatever nixpkgs they want) to keep buck2 stable, but upstream nixpkgs itself will want some stable releases on a regular cadence, for regular usage, and so that other packages inside nixpkgs can eventually use buck2 itself. Other distros will eventually want buck2 packages for the same reasons.