search

nix-community / rnix-parser

A Nix parser written in Rust [maintainer=@oberblastmeister]
MIT License
365 stars 44 forks source link

fix: discrepancy on leading 0's with cpp nix's behavior. #160

Open hsjobeki opened 1 year ago

hsjobeki commented 1 year ago

Summary & Motivation

As described in #157, Nix doesn't allow additional leading 0's on floats. Adding additional leading zeros to numbers may even break them into two tokens. (TOKEN_INT and TOKEN_FLOAT) I've also described further information in the referenced issue.

This means if we closely follow c++ Nix's behavior, with this PR, rnix tokenizes as follows. (Which represents Nix's behavior)

01.1e2 -> 

TOKEN_INTEGER, "01"
TOKEN_FLOAT, ".1e2"

01.e2
->

 TOKEN_INTEGER, "01"
TOKEN_DOT, "."
TOKEN_IDENT, "e2"

This may fail to parse/evaluate, but we don't care at the tokenizer level.

nix-repl also shows that Nix tokenizes the same way.

nix-repl> 01.e2
error: value is an integer while a set was expected

@tazjin The error here may need improvement. But we can handle that further, e.g., in the tvix compiler. To give better error messages than C++ nix.

Backwards-incompatible changes

All tests on floats were run successfully. Since this only adds custom handling on leading 0's, this doesn't break any backward compatibility.

Further context

Solves #157

hsjobeki commented 1 year ago

This PR can be closed. As soon as https://github.com/NixOS/nix/issues/8605 is fixed, we don't have any discrepancies. rnix tokenized correctly already.