Open loafofpiecrust opened 2 years ago
FWIW, I would love Rust interop with the Nix C++ codebase. I am very much in your boat and find my time in Rust much more productive. There may even been cases where Rust could make our lives a lot easier.
In the longer term, a Rust API to the Nix daemon would also be really interesting :thinking:
I might be able to craft an initial PR hooking it up.
Would maintainers be willing to support a PR of this nature? Further, would the project be open to future PRs in Rust for new > features or rewriting existing code if it's significantly simpler?
Not speaking in anyone’s name except my own (esp. given that I wasn’t involved in the previous Rust experiment), but I think the main reason why the previous experiment got abandoned was that it was adding a lot of complexity while not providing a lot of benefits (IIRC the only part that was written in rust was the StorePath
abstraction, which was arguably much nicer in rust, but was also pretty small, so not really worth all the trouble).
So imho, Rust contributions could be totally acceptable (and even great, I for one would be happy to be able to write Rust code rather than c++), provided that
nix-rust
setup (in terms of build-system in particular, I think I remember that was a bit of a pain)I'd like to collect previous efforts on the nix-rust partial-port side (the further down the list the more it deviates from nix):
cc @griff
Hey, I'm also trying to start something (https://github.com/ldesgoui/rust-in-nix), I wanted to prepare a little intro to give context, but I don't have the time immediately, the gist of what I'd like to achieve is in the README. Happy to get in touch over Matrix
As it appears that there are now at least 4 tries to reimplement parts of Nix in Rust, it might be appropriate to isolate common functionality or patterns into crates that can be shared between all of them (as it might be likely that we run into edge cases which could be modeled with a big variety). It is also easy to see that the initial development targets of the associated projects vary to a somewhat greater degree. Maybe I should create a GitHub org to organize and bundle these efforts. (especially because I don't want to litter this issue thread with meta-talk)
edit: created https://github.com/RIIR-Nix
@ldesgoui
Happy to get in touch over Matrix
what's your Matrix identifier (username and server)? (mine's @zseri:matrix.org
)
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/the-uncompromising-nix-code-formatter/17385/39
I noticed that Flex and GNU bison with %glr-parser
support as required by Nix ambiguous grammar, used in the Nix lexer and parser, do not have an equivalent in rust. There is lalrpop, but that is.. LALR, and Nix requires GLR or Earley (since it is an ambiguous grammar)
So I just created a library for it, and used it to port the lexing and parsing components to Rust (source here)
From this to having a Nix evaluator written in Rust we are very very close: just visit each node and perform the small set of actions in parser.y to build the symbol table, etc
The result is easier to work with than the old nix-rust setup (in terms of build-system in particular, I think I remember that was a bit of a pain)
There’s some real and consistent benefit from it (maybe some kind of plan on how to migrate a substantial part of the codebase to Rust as that’ll probably not be something that will happen overnight nor spontaneously)
I suppose the advantages derive from this being written in rust:
Since we would be using something based on Flex and GNU Bison, we reduce the possible bugs due to the simple fact of changing the lexing and parsing engine, and worst, the bugs that would be created by having to change the grammar, actions, or lexing states to accommodate ourselves to something else than flex-bison
A small demo:
echo 'let a = 123; in a'| cargo run --example nix
Lexemes:
LET "let" (1, 1)
ID "a" (1, 5)
= "=" (1, 7)
INT "123" (1, 9)
; ";" (1, 12)
IN "in" (1, 14)
ID "a" (1, 17)
AST:
Γ
expr
expr_function
LET "let" (1, 1)
binds
binds
attrpath
attr
ID "a" (1, 5)
= "=" (1, 7)
expr
expr_function
expr_if
expr_op
expr_app
expr_select
expr_simple
INT "123" (1, 9)
; ";" (1, 12)
IN "in" (1, 14)
expr_function
expr_if
expr_op
expr_app
expr_select
expr_simple
ID "a" (1, 17)
Santiago is still a work in progress, for instance it is missing Bison's %left %right and %nonassoc statements, and instead it returns all possible ASTs according to the ambiguous grammar, but that should be simple to implement, and anyway is a very relevant library to this issue
I got the Nix AST in rust: https://github.com/kamadorueda/nixel
From this point, if we wanted to have a Nix evaluator in Rust we would need to add the EvalState abstraction, build the symbols table, the built-ins and the "Thunk" abstraction to evaluate stuff lazily
The imo hardest builtin to correctly implement/"find" is builtins.match
, because it uses the C++ regex engine, and it is not simple to replicate it because every regex engine seems to use slightly different syntax. It's doable, sure, but probably takes some effort to port or wrap (as in: providing a simple rust wrapper which basically links against the snippet...)
@zseri One step at a time
Just mentioned it because besides https://github.com/YZITE/nix2js/issues/2, in nix2js
that was an important problem, and the standard rust regex
crate does afaik use a different syntax than the one expected by builtins.match
.
Okay, so if someone wonders what a Nix evaluator in Rust would actually look like, this would be a good example: https://github.com/kamadorueda/toros
Currently able to evaluate things like:
let
add = a: b: a + b;
in
add 1 2
Which involves complex things like scopes, built-ins, currying, identifiers and laziness
Probably the tests folder is a good place to see the current capabilities
I also saw a few interesting possibilities, like compiling to Web-Assembly, embedding into Jupyter notebooks, and defining the store interface as a trait so that people can bring their own amazing back-end (s3? ipfs? dna?)
Rename Rc
to Arc
and boom, parallel evaluation in the corner. Hash of input file and boom, file-level cache. And so on
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1
Any updates on this?
See what the Tvix people are up to.
@Ericson2314 I thought Tvix wasn't meant as a replacement to Nix, is there any chance in the future that people will be able to contribute to Nix in Rust?
@Nyabinary, the stated goal in the material I read, at least, was to create an Nix evaluator that was capable, eventually of course, of evaluating all of nixpkgs, so in that sense it should definitely be usable as a full replacement (once it's ready).
@nrdxp Would a gradual replacement be possible maybe? Also, a Matrix room for this effort would be really nice to have.
@Nyabinary if you want to stay up to date with Tvix, you can join the IRC channel through Matrix.
A gradual replacement is very possible. I have done lots of work to make the daemon protocol be better documented. Now, I really hope tvix will implement the daemon protocol. Then we can mix and match stores/evaluators from either project.
A gradual replacement is very possible. I have done lots of work to make the daemon protocol be better documented. Now, I really hope tvix will implement the daemon protocol. Then we can mix and match stores/evaluators from either project.
That would be absolutely great, considering tvix does some very smart things in their reimplementation.
Exactly!
I want to contribute to Nix and the surrounding ecosystem, but I find that modifying existing code or writing new code in C++ is much more error-prone for me and I (along with others, I'm sure) could give much more meaningful contributions in Rust.
I've noticed overlap in the Rust and Nix communities and I'm sure that having a path to contribution here in Rust would energize people and Rust anyway fits really well with the Nix idea of static guarantees about a system.
I see that there's a nix-rust folder where someone attempted to port a piece of Nix to rust, but this project was abandoned. It seems like there's solid C++-Rust interop now with the cxx project, and I might be able to craft an initial PR hooking it up.
Would maintainers be willing to support a PR of this nature? Further, would the project be open to future PRs in Rust for new features or rewriting existing code if it's significantly simpler?