Feedback

[bd82]

These are all suggestions :)

• The main description is a little unclear to me.

  • What I see here is: "an in Memory LL(K) Parser Generator implemented using Chevrotain"

• Link to a wiki or some spec of EBNF in main readme? e.g: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

• Link to Chevrotain in main readme?

• Add an ebnf snippet in main readme code boxes?

• Specify limitation with regards to Content Security Policy due to using eval?

  • https://sap.github.io/chevrotain/docs/guide/custom_apis.html#code-generation

• Evaluate exposing a source generation API as well to avoid eval, and allow easier debugging.

  • So it won't only be "in memory"

• Link to Chevrotain CST documents at the end of the code samples (When a CST is created).

[bd82]

let lexer = new Lexer(Object.values(parser.tokensMap));

The order of tokens is meaningful in some cases.

• https://github.com/SAP/chevrotain/blob/master/examples/lexer/keywords_vs_identifiers/keywords_vs_identifiers.js
• Or integer vs Decimal

So:

let lexer = new Lexer(Object.values(parser.tokensMap));

May be a bit naive. 😢

[rdking]

let lexer = new Lexer(Object.values(parser.tokensMap));

The order of tokens is meaningful in some cases. ... May be a bit naive. 😢

It's painfully naive. The next version will probably ask for an ordered token map. That way I can give the developer a chance to both order and name the tokens. The current naming pattern is also fairly unfortunate. 😓

[rdking]

Evaluate exposing a source generation API as well to avoid eval, and allow easier debugging.

I wrote that feature in already. I just haven't enabled it yet because I was busy testing to get the core logic working. I wanted to wait until I solved the token order issue before I released that feature.

[bd82]

I wrote that feature in already. I just haven't enabled it yet because I was busy testing to get the core logic working. I wanted to wait until I solved the token order issue before I released that feature.

No rush. 👍

You may be able to perform some analysis on the regular expressions I do the same in Chevrotain for optimization purposes using this library

• https://github.com/bd82/regexp-to-ast

I think there may be other libraries that convert a regexp to an AST for analysis too... Also it is naturally generally safe to put longer tokens before shorter ones. If you can solve the ordering problem, we could use that logic to perform validations on Chevrotain lexers (hand written).

I think this would be related to equivalence of two finite state automaton.

[rdking]

I noticed something I thought a bit peculiar. With Chevrotain no token string is ever allowed to match against more than 1 token, even if the grammar makes the use cases unambiguous. Is there any particular reason for that?

[bd82]

I noticed something I thought a bit peculiar. With Chevrotain no token string is ever allowed to match against more than 1 token, even if the grammar makes the use cases unambiguous. Is there any particular reason for that?

That is because Chevrotain uses a separate and distinct lexing phase so it cannot use grammar context to decide between lexing options.

It should be possible to do lexing on demand particularly if/when a streaming lexer would be implemented

• https://github.com/SAP/chevrotain/issues/845
• I created a (hack) lexing on demand prototype once, so it is definitively possible.

But that does not exist as a feature currently...

[rdking]

Ok. I just released v1.1.1. This includes the ability to specify names and ordering for tokens as well as the ability to generate source code. I've up dated the documentation, also. If you see any room for improvement, feel free to submit a patch.

[bd82]

No extra feedback at this time.

I've linked your project from the custom API example docs.. https://github.com/SAP/chevrotain/blob/master/docs/guide/custom_apis.md#runnable-examples