Saves ~75 lines of code and makes it a lot easier to add a new token. Previously, adding a new token could mean you'd have to make changes three places in the code: enum definition, string representation table and keyword hashmap (in addition to any lexing and parsing rules). Now its only a matter of adding it to the appropriate X macro list. The downsides is this relies heavily on the preprocessor and makes the code more complicated to read and understand if you are not familiar with X macros.
Changes:
In Short:
Less jank. Less lines.
X macro for TokenType
Saves ~75 lines of code and makes it a lot easier to add a new token. Previously, adding a new token could mean you'd have to make changes three places in the code: enum definition, string representation table and keyword hashmap (in addition to any lexing and parsing rules). Now its only a matter of adding it to the appropriate X macro list. The downsides is this relies heavily on the preprocessor and makes the code more complicated to read and understand if you are not familiar with X macros.
New TokenType values
@[, ', ;, dt_list, dt_tuple, dt_map, t_break, t_continue
Added line and column information to lexer and individual Token
Fixed the lexing of:
optional '_' as a visual separator for large numbers
strtodimplementation.Updated the map and tuple syntax
@[ "key": "value" ]'1, 2, 3'And some minor bug fixes