Playground error message - question

[brucou]

Hi,

I have an issue that appeared with the playground. It may be just my misakes, but the error message displayed there is not insightful and I can't progress further after quiet the time, so I am posting here. I looked for a Stack-overflow tag but I could not find one, Please let me know if there is a better place to ask these things, as I see from the current issues that there arent really such help-me question there.

Alright so here, we go. The code in the playground is as follows:

(function jsonGrammarOnlyExample() {
  // ----------------- Lexer -----------------
  const createToken = chevrotain.createToken;
  const Lexer = chevrotain.Lexer;
  const CstParser = chevrotain.CstParser;

  const LSquare = createToken({name: "LSquare", pattern: /\[/});
  const RSquare = createToken({name: "RSquare", pattern: /]/});
  const Slash = createToken({name: "Slash", pattern: /\//});
  const Comma = createToken({name: "Comma", pattern: /,/});
  const MetaChr = createToken({name: "MetaChr", pattern: /[,!@#$%^&*()"':{}=+-;]+/});
  const StringLiteral = createToken({
    name: "StringLiteral", pattern: /(:?[^\\"\n\r]+|\\(:?[bfnrtv"\\/]|u[0-9a-fA-F]{4}))*/
  });
  const NumberLiteral = createToken({
    name: "NumberLiteral", pattern: /-?(0|[1-9]\d*)(\.\d+)?([eE][+-]?\d+)?/
  });
  const WhiteSpace = createToken({
    name: "WhiteSpace",
    pattern: /\s+/,
    group: Lexer.SKIPPED
  });

  const tokens = [WhiteSpace, NumberLiteral, StringLiteral, LSquare, RSquare, Comma, Slash, MetaChr];

  const yedLexer = new Lexer(tokens, {
    // Less position info tracked, reduces verbosity of the playground output.
    positionTracking: "onlyStart"
  });

  // Labels only affect error messages and Diagrams.
  LSquare.LABEL = "'['";
  RSquare.LABEL = "']'";
  Comma.LABEL = "','";
  Slash.LABEL = "/";

  // ----------------- parser -----------------
  // eventClause
  // : StringLiteral+
  // guardClause
  // : StringLiteral*
  // actionsClause
  // : StringLiteral*
  // edgeClause:
  // [eventClause] [\[ guardClause \]] [/ actionsClause] |

  class Parser extends CstParser {
    constructor() {
      super(tokens, {
        recoveryEnabled: false
      })

      const $ = this;

      $.RULE("yed", () => {
        $.OPTION1(() => {
          $.SUBRULE($.eventClause);
        })
        $.OPTION2(() => {
          $.CONSUME(LSquare);
          $.SUBRULE($.guardClause);
          $.CONSUME(RSquare);
        })
        $.OPTION3(() => {
          $.CONSUME(Slash);
          $.SUBRULE($.actionsClause);
        })
      });

      $.RULE("eventClause", () => {
        $.MANY(() => {
          $.CONSUME(StringLiteral)
        })
      });

      $.RULE("guardClause", () => {
        $.MANY(() => {
          $.CONSUME(StringLiteral)
        })
      });

      $.RULE("actionsClause", () => {
        $.MANY(() => {
          $.CONSUME(StringLiteral)
        })
      });

      // very important to call this after all the rules have been setup.
      // otherwise the parser may not work correctly as it will lack information
      // derived from the self analysis.
      this.performSelfAnalysis();

    }
  }

  // wrapping it all together
  // reuse the same parser instance.
  const parser = new Parser([]);

  // ----------------- Interpreter -----------------
  const BaseCstVisitor = parser.getBaseCstVisitorConstructor()

  class yedInterpreter extends BaseCstVisitor {

    constructor() {
      super(tokens, {recoveryEnabled: false})
      // This helper will detect any missing or redundant methods on this visitor
      this.validateVisitor()
    }

    yed (ctx) {
      return {event: this.visit(ctx.eventClause), guard: this.visit(ctx.guardClause), action: this.visit(ctx.actionsClause)}
    }

    eventClause (ctx){ return ctx.StringLiteral ? ctx.StringLiteral : ""}
    guardClause (ctx){ return ctx.StringLiteral ? ctx.StringLiteral : ""}
    actionsClause (ctx){ return ctx.StringLiteral ? ctx.StringLiteral : ""}
  }

  // for the playground to work the returned object must contain these fields
  return {
    lexer: yedLexer,
    parser: Parser,
    visitor: yedInterpreter,
    defaultRule: "yed"
  };
}())

The input I pass is m m m m / a.

The error message I get is:

Errors during evaluation of the implementation: 
Cannot read property 'PATTERN' of undefined

If that matters, running on Windows in the Brave browser (Chromium 81).

The language I want is fairly simple: a [b] / c for instance shoud be recognized and a, b, c can all be empty strings too. For the STRING_LITERAL, I reused the JSON_LITERAL and removed the starting and end quotes ". When doing that, I get the aforementioned error. If you put the quotes back, the error is no longer but then the string literals are not parsed (because they do not start with a quote).

Any ideas what I can be doing wrong? I don't see which PATTERN Chevrotain is trying and failing at validating.

[bd82]

Hello @brucou I am unable to identify the issue.

The playground is not meant to be used as a development environment It is just a little toy to demonstrate Chevrotain

So its not something I would invest in exploring/debugging/fixing...

[bd82]

If you prefer coding inside a browser maybe try one of:

• https://visualstudio.microsoft.com/services/visual-studio-codespaces/
• https://www.gitpod.io

[brucou]

It is my first time using and evaluating Chevrotain, so the playground and starting from working examples looked like the easiest way to try things out with a simple grammar like the one I have.

Error handling is hard, and often neglected in most open source projects (just like documentation). In the time I took to implement this small example from scratch from the docs and with no prior knowledge, the truth is the playground and its immediate feedback and helpful error messages was quite helpful. I had the ambiguity messages and a link which helped solve the problem. That's awesome! A pity it stopped being useful. It is a pretty good tool for starters.

Anyways, I will try on my local machine then (Thanks for your gitpod recommendation though). Maybe the error disappear or will be more helpful. Thanks! I do understand that this is the place to write about issues with grammars then? Or do you have another channel like stack overflow for that? That trivial grammar I have there is just a minimal piece of a textual language I want to develop to describe behaviour of UIs through state machines, so unavoidably I expect to reach further hurdles when I complexify things...

[bd82]

Error handling is hard, and often neglected in most open source projects (just like documentation). In the time I took to implement this small example from scratch from the docs and with no prior knowledge, the truth is the playground and its immediate feedback and helpful error messages was quite helpful. I had the ambiguity messages and a link which helped solve the problem. That's awesome! A pity it stopped being useful. It is a pretty good tool for starters.

Maybe there should be a quick start template. Sort of like a playground but as a npm package with:

• Simple Grammar
• Script to generate diagrams
• Input file
• Script to parse the input and serialize it to some JSON output.
• ???

Basically the flow of the playground but in a real dev env.

The playground is just a simple naive hack not a fully fledged dev env.

[bd82]

I do understand that this is the place to write about issues with grammars then? Or do you have another channel like stack overflow for that?

There is a gitter chat channel linked from the docs website / README.md of the repo.

[brucou]

@bd82

Maybe there should be a quick start template.

That would be awesome!! Yes, the main issues for beginners are:

• is that the right syntax?
  • that is covered by docs, the javascript IDE, and also error messages (I would put those ambiguity warnings in that category?)
• is that the right semantics?
  • this is where you pass inputs and look at the outputs. There are two problems:
  • when there are no outputs: why??
  • when there are incorrect outputs: why??

There is a gitter chat channel linked from the docs website / README.md of the repo.

Great, thanks, I missed that information apparently. Just realized that the link to the chat is this small badge in that badge bar that I always skip -- usually it is build status, size etc not what I look for when I open the README the first time. Maybe adding an explicit link would also help.

[brsanthu]

@bd82 created this starter pack which does following things.

1. lexer
2. parser
3. visitor
4. serializer
5. diagram generation
6. unit tests

Please let me know if you have any feedback.

https://github.com/datasert/chevrotain-starter/pull/1/files

[brucou]

@brsanthu that's awesome:

• I loved the commented code. For instance: // "input" is a setter which will reset the parser's state. I had not realized that the parser was stateful and needed to be reset to parse another string. That is typically the small things where you can spend hours trying to figure what the hell is going on.
• those comments too are useful
• I think the visitor is fairly clear. Not sure what the serializer is about though. Oh I see you need the serialized grammar to creae the diagram,, fair enough.

Then, the way I go about writing this language is by writing a tiny piece of it, validate the parsing of that, and then improve the grammar iteratively. So I guess I will have to have some tests that will grow as the grammar will grow. The tests I see are clear and give an idea of how to structure them.

Then there is the stage when one test fails and where it is necessary to figure out what is wrong. I don't know if it is possible to see what rules have been analyzed, i.e. trace the parser execution.

Anyways, I will have a deeper look at it tomorrow but it looks great. Thank you so much guys.

[brsanthu]

Parser converts text to model and serializer converts model to text. To generate diagram, you just need parser.

Those comments are from the Chevrotain tutorial.

[bd82]

created this starter pack which does following things.

Thanks @brsanthu I think this can be very useful.

Are you interested in contributing it to this repo:

• https://github.com/SAP/chevrotain/tree/master/examples
• we can create a new quick-start folder

After a quick look my feedback (assuming contribution to Chevrotain) is:

• Be un-opinionated and focus on the chevrotain parts

  • Only use minimal default tsconfig.
  • avoid eslint.
  • avoid prettier (its part of the chevrotain mono repo anyhow).

• Add more in-depth usage instructions at the top level readme

  • I can help with that.

I will attempt to dive into this in more depth soon.

[brsanthu]

@bd82 would be more than happy to contribute so it can live as part chevrotain repo. Couple of things.

1. I feel it would be much easier for somebody to get going if that started pack is separate repo than part of this mono repo. Examples are meant to be referenced but starter packs are meant cloned as whole, change few things and get going with their work. So sap/chevrotain-starter may be better. Also if it is separate repo, it is easier to configure to the level that is actually better as consumer of chvrotain than developers of chevrotian. Below point would be connected this thought as well.

2. About eslint and prettier: many times, users will just get going with what comes with something by default and not try to pay attention details of config, especially peripheral config like linter/styles. So I feel that configuring those for best quality, would help users adopt the best practice.

3. If you think, having typescript as the lang in the starter, we could think about creating chevrotain-starter-ts and chevrotain-starter-js repos so users can pick the one they want.

4. About docs, yes, we need to enhance it.

[bd82]

Hi @brsanthu

Sorry for the late response.

MonoRepo vs Separate repo.

I agree a separate repo is easier for consumers, but there is also the concern of testing the quick-start repo. If its part of the mono repo it would be ensured to be updated (e.g on breaking changes).

Is there some way to have the cake and eat it too?

• e.g a git clone of a specific sub folder of a repo?
• Some kind of nesting of git repos (git sub-modules?)

Dev Flows configurations (linter/formatter).

This is quite problematic.

• Some tools choice are opinionated, Who is to say X is better than Y tool?
• Secondly I'd rather not maintain the dev-flow tools of a quick start repo, e.g:
  • In the past TSLint was used, but now its deprecated.
  • Updating versions of dep deps and breaking changes.

IMHO The scope has to be minimal and focused on Chevrotain flows, not generic dev flows.

Multiple QuickStarts

Both TypeScript and JavaScript quick starts are a good idea

[bd82]

The Chevrotain repo should move to a new organization in the future. I've created a quick-start repo there you can contribute to if you want to start with a separate repo flow.

• https://github.com/Chevrotain/quick-start

With some luck I'll be able to connect it to the mono-repos CI flow somehow...

[bd82]

@brsanthu I've sent you an invite for write permissions in https://github.com/Chevrotain/quick-start

[brsanthu]

Added the files from earlier repo as this PR. https://github.com/Chevrotain/quick-start/pull/1/files

Removed the eslint and prettier config from the setup. compile, test and diagram generation works fine.

[bd82]

Thanks @brsanthu

I've done a quick review, lets continue the discussions on that repo. I will close this issue for now.

You can also send me a direct message on gitter (link in the root README)