parse tokens incrementally; kill a grammar ambiguity

[mathisonian]

• Please check if the PR fulfills these requirements

This is an internal change to improve the performance of the compiler, and prepare for some upcoming changes for better IDE support and error messages.

• [x] Tests for the changes have been added (for bug fixes / features)

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

This should not result in any external behavioral change. It changes how the parser is invoked internally. It also changes the grammar to remove a potential ambiguity (but does not change the parsed output).

• What is the current behavior? (You can also link to an open issue here)

Some files fail to render (e.g. https://github.com/idyll-lang/idyll/issues/736) due to memory usage.

• What is the new behavior (if this is a feature change)?

Files are properly parsed and rendered.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No.

[mathisonian]

/cc @TwoTau. To get this to work I had to change the return value of the lexer rules. In particular, they are now wrapped in another array. This will change further as we add additional metadata to the tokens themselves, but I want to get this incremental change in quickly since it fixes some issue.