misparsing floats/ints starting with zeroes

[yorickvP]

Describe the bug

rnix-parser does not parse floats/ints the same as the nix official parser.

Code Snippet to reproduce

let e="e"; in [001.22e01e.30.4]

(yes, this is valid nix)

Gets parsed into TOKEN_FLOAT@18..27 "001.22e01"

Expected behavior

semantically equivalent to [ 1 2.2 "e" 0.3 0.4 ]

Additional context

[tazjin]

I think the bug appears in a smaller snippet already. If we remove the e situation:

nix-repl> [001.22e01.30.4]  
[ 1 2.2 0.3 0.4 ]

but in Tvix (using rnix):

> [001.22e01.30.4]
[ 12.2 0.3 0.4 ]

(AST is dumped on that link)

[yorickvP]

This is how I fixed it in nixfmt: https://github.com/serokell/nixfmt/commit/a45763a277fe048386d94a6481123828e574b695

[hsjobeki]

This error happens at the tokenizer level already. But it is questionable if this is really an error and not a mistake in nix itself (as i will question)

001.22e01.30.4

gives the following rnix tokens

TOKEN_FLOAT, "001.22e01"
TOKEN_FLOAT, ".30"
TOKEN_FLOAT, ".4"

In contrast to that, It seems nix itself disallows leading 0 on floats in general

01.1e2 -> error: attempt to call something which is not a function but an integer 01.1 -> error: attempt to call something which is not a function but an integer but 1.1e2 -> 110 11.1e2 -> 1110

leading number chars != 0 are no problem in nix floats it seems.

I'd like to know what IEEE says about this. But I couldn't find if this is even specified. Maybe @RaitoBezarius knows where to look at?

To describe nix's behavior:

0 Chars, must be:

• either of an Integer OR
• immediately preceded by an .(DOT) which makes them part of a float. The remaining chars must then tokenize into a valid float.

Implementing this into rnix should solve this issue I think. (I will come back to this when I got some time)

[hsjobeki]

Update: it seem ieee doesnt specify leading 0‘s its the responsibility of the app to ignore leading 0‘s. So in summary this is a bug in the nix tokenizer. rnix is right, but doesnt follow nix‘s broken behavior. It may be up to the maintainer (@oberblastmeister ) to decide whether rnix should follow this logic.

I‘ll open an issue for this in nix.

[RGBCube]

Update: it seem ieee doesnt specify leading 0‘s its the responsibility of the app to ignore leading 0‘s. So in summary this is a bug in the nix tokenizer. rnix is right, but doesnt follow nix‘s broken behavior. It may be up to the maintainer (@oberblastmeister ) to decide whether rnix should follow this logic.

I‘ll open an issue for this in nix.

Do you have a link to the issue?