Can we use a newer shell?

[thoughtpolice]

The rules written from scratch here don't have to use bash, and alternative shells such as nushell are a possibility. The questions are:

• Can we use this shell for scripting in rules?
  • The workflow is something like "Write a script file then run nushell on it to do something arbitrary" via ctx.actions.run
  • The mother of all these things is genrule()
  • It would always be good to not use Bash for anything important.
• Can we use it for the development environment in a turnkey, reasonable way?
  • That is, the user should get dropped into a pre-configured shell, just like today
  • We could utilize this to have a much more uniform experience.
  • This is where a better language will really shine, since we can equip the environment with more tools and utilies.

Currently, I have used Oil as a replacement for Bash in this repository, with the hope that we would be able to use the Oil language. But inevitably some other platform compatibility may occur for some subset of the rules, and having a single shell language would be great. There's no reason to tie ourselves to something trying to be compatible and besides, the C++ implementation of Oil is a ways off yet.

Powershell is also another possibility, actually. But I think those are the two (or three) main possible contenders.

[saolof]

An opinion from a random person checking out the repository:

IMO it depends on what the scripts you write do. Posix compliant shell isn't that bad of an option as long as the script is reasonably simple (just creating pipes between programs and setting environment variables) and has lots of implementations of varying quality (dash, bash, zsh, ysh, yash, etc etc).

Posix has it's downsides though. I personally really like elvish (clojure inspired shell) but while it has potential I would not consider it mature enough for production usage. Imho, if you specifically want a non-posix shell for scripting, nu does look like the most solid alternative:

• Excellent type checking and error messages
• Written in a memory safe language
• Less heavyweight than Powershell
• Apart from having >20k stars, the Github repository has at least ten people who have contributed more than 10k sloc over a longer time period, and several maintainers can merge commits to the repository, which is very much not the case for smaller projects which have significant project risk from a single person disappearing

Another option may be to use a non-shell programming language with good scripting support, if the task is a script does not have to be in a shell per se.

[thoughtpolice]

Thanks for the input, I hope this codebase has taught you something or been interesting to read. The key points about nushell's viability wrt its active development/low bus-factor are also really good.

So, as an update, as an alternative to direct genrule()s, for a (currently unreleased) fork of this prelude, after some searching — I've been using https://github.com/babashka/babashka in order to actually write real "scripts" that aren't in a shell language and that are run for various tasks. This has been pretty good for small scale stuff in my experience. For example, here is a target determinator (that assumes jujutsu as the underlying VCS) in order to compute the changed buck2 targets between two revisions: https://gist.github.com/thoughtpolice/4f3e1b89179125302266bbf51db673c2

austin@GANON$ buck2 -v0 run tools//babashka:bin -- ./determinator.bb --from @-
Build ID: fef5ec60-57d7-42c5-bca3-d250baa8746b
Network: Up: 0B  Down: 0B
Jobs completed: 3. Time elapsed: 0.0s.
Performing target determination, from base to given:
 ├─  base: work/td-ia0v909evbeyf ⇜  @-
 └─ given: work/td-ia0v909evbeyt ⇜  @

Performing target hash comparison between workspaces...
Differences DETECTED:
   ├── DELETED: 0
   ├─┬ MODIFIED: 8
   │ ├─⇝  tilde//aseipp/libedsign:lib
   │ ├─⇝  tilde//aseipp/libedsign:test-roundtrip.exe
   │ ├─⇝  tilde//aseipp/libedsign:test-roundtrip
   │ ├─⇝  tilde//aseipp/libedsign:test-fingerprint.exe
   │ ├─⇝  tilde//aseipp/libedsign:test-fingerprint
   │ ├─⇝  tilde//aseipp/libedsign:test-rekey
   │ ├─⇝  tilde//aseipp/libedsign:test-rekey.exe
   │ └─⇝  root//:libedsign
   └── ADDED: 0

Removing workspace 'td-ia0v909evbeyf'
 ├─ `jj abandon qpwllkqopmormrssspvmkvqovypxvtrq`
 └─ Deleting workspace tree work/td-ia0v909evbeyf
Removing workspace 'td-ia0v909evbeyt'
 ├─ `jj abandon tonwttnkttuksztzvsktyyxzrrpmnrps`
 └─ Deleting workspace tree work/td-ia0v909evbeyt
Done.

There were some technical considerations that drove the use of Babashka, most importantly:

• I wanted it to work on multiple platforms INCLUDING windows (though Windows is not a target for this repository, it is planned for another project I'm working on)
• It was self-contained
  • This meant it could be used directly inside the prelude to run scripts and automation at the 'lowest level', without a complicated ruleset
  • It had a statically available Linux binary, meaning it could circumvent any need to have a bootstrapped glibc. Again, this makes usage at the 'lowest levels' of a Prelude a lot easier.
• Lisp is nice. I think Lisp is nice, anyway. The fact is that most of the scripts will be relatively small self-contained programs, so they can remain pretty readable and usable. (I tend to write fairly 'first order' lisp programs, frankly.)

I felt Babashka (or something like it) was a better choice than say, Python IMO, for a few reasons, whereas it is the tool of choice in the upstream buck2-prelude (hard to statically link, imports are a bit of a mess, the stdlib is fairly complete but has some misses and including 3rd party stuff sucks). You might notice that, except for the last point, the same points for Babashka hold for nushell too — Windows is a major target, self contained, statically linked on Linux.

---

The main UX tradeoff I see here is the question: do you want to write the script inside the BUILD file, or in a separate source file? I feel like the first case is where the brevity of a "shell language" like nushell would really shine through, while in the second case you probably want something like Babashaka (or Elvish, or whatever your favorite choice of Lisp is.)

So, after working more with all this a bit, my basic opinion is probably (as you might expect) now forming and molding into something like:

• If you want to write real genrule()s that exist inside of BUILD files, you want the brevity of a shell language.
  • For the same reasons you mentioned, among others (including the ones above), I think nushell is probably the right choice.
• For out of band scripting that is too large to fit in a BUILD file, you want a self contained scripting language, such as Babashka or really whatever you like. I mean, python is fine too if you can get a self-contained and statically linked portable image.