numeric literals allow '_' as separator

[JLPLabs]

Many languages allow the use of '_' as separator in numeric literals. [1] [2] [3] [4] [5] Allowing such a separator makes it easy to check your work, visually (e.g., distinguish words in hexadecimal and 000's in decimal). local x = 0xCAFE_F00D local y = 1_000_000

To accomplish this required telling the lexer to accept '_'.

• lexdecimals[""] = true
• lexhexadecimals[""] = true

And then to strip it out when ending a token:

• a helper function strip_under()
• calling helper from 3 "end_token" functions

Example:

$ cat 42_000.tl

local t1 = 42_000
local t2 = 42_000.0
local t3 = 42_
local t4 = 42_000.000_123
local t5 = 1_000E3
local t6 = 0xCAFE_F00D

print(t1)
print(t2)
print(t3)
print(t4)
print(t5)
print(t6)

$ tl run 42_000.tl

42000
42000.0
42
42000.000123
1000000.0
3405705229

Feel free to use this or to ignore it if it isn't high on your priority list. -John

[1] rust -- https://doc.rust-lang.org/rust-by-example/primitives/literals.html [2] C# -- https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-7.2/leading-separator [3] python -- https://www.python.org/dev/peps/pep-0515/ [4] java -- https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores-literals.html [5] php -- https://wiki.php.net/rfc/numeric_literal_separator

[github-actions[bot]]

Teal Playground URL: https://447--teal-playground.netlify.app

[JLPLabs]

I see travis is failing on:

Point.__index = Point

I'll look into this!

[JLPLabs]

note: you will see some of my comments still in the code --JRP: If you think my work is acceptable then I'll go back and strip those out.

[hishamhm]

Hi @JLPLabs! The patch itself looks correct (though the added checks on every end_token would add some unnecessary cost on a very hot part of the code, and this could be implemented as a state machine transition that only affects the number rules), but I think we shouldn't add this feature because, at least at this point in time, we are avoiding adding features to Teal which are syntactic changes to Lua which are unrelated to type checking.

I'm closing this one if you don't mind, but I hope the dive into the Teal lexer was a fun exercise in any case!

[JLPLabs]

That makes perfect sense to me. I did find it an interesting problem to solve.