kach / nearley

📜🔜🌲 Simple, fast, powerful parser toolkit for JavaScript.
https://nearley.js.org
MIT License
3.6k stars 232 forks source link

Parser compilation fails if token names collide with JavaScript keywords #525

Open TheGrandmother opened 4 years ago

TheGrandmother commented 4 years ago

I'm rewriting the grammar for my language to use a lexer (:cow:) and i run in to this massively annoying problem.

So I had this:

compound ->
    assignment
  | function_call
  | "return" _ expr                                                           {% ast.makeReturn %}

And changed it to:

compound ->
    assignment
  | function_call
  | %return _ expr                                                           {% ast.makeReturn %}

Where %return is a token from the lexer.

I then got this error:

SyntaxError: /home/grandmother/git/Changtopia/changtopia/changlang/compiled_grammar.js: Unexpected token (41:79)

 39 |     {"name": "compound", "symbols": ["assignment"]},
  40 |     {"name": "compound", "symbols": ["function_call"]},
> 41 |     {"name": "compound", "symbols": [(lexer.has("return") ? {type: "return"} : return), "_", "expr"], "postprocess": ast.makeReturn},

As you can se it has taken %return and just used return as the name of the variable in the parser.

The same appears to be happening for any token name that collides with a JavaScript keyword.

Is the only solution to this to just name my tokens differently?

I'm rolling with: Node version: 13.14.0 Nearley: 2.19.3 moo: 0.5.1