kach / nearley

📜🔜🌲 Simple, fast, powerful parser toolkit for JavaScript.
https://nearley.js.org
MIT License
3.57k stars 231 forks source link

Can't undertsand indentifier token #623

Closed NovaAppsInc closed 1 year ago

NovaAppsInc commented 1 year ago

I'm using the latest moo lexer and nearley

right now I'm testing out these tokens: identifiers, assignment operators, and literals with this line below

n--4

in the lexer I have these set as tokens in lexer.js

module.exports = moo.compile({
    keyword: ['allow', 'make', 'process', 'is', 'else', 'for'],
    indentifier: /[a-zA-Z_%][a-zA-Z0-9_%]*/,
    whitespace: /[ \t]+/,
    assignment_op: "--",
    number: { match: /0|[1-9][0-9]*/, value: Number },
    string: /"(?:\\["\\]|[^\n"\\])*"/,
    seperator: ':-',
    lparen: '(',
    rparen: ')',
    newline: { match: /\n/, lineBreaks: true },
    codebreak: { match: /;/, lineBreaks: true },
    lineend: { match: /\r\n/, lineBreaks: true },
});

nearley grammar in grammar.ne

@{%
const yaklingLexer = require('./lexer');
%}

@lexer yaklingLexer

assignment -> %indendifier %assignment_op literal

literal
    -> %number
    |  %string

made a parser.js

const nearley = require('nearley');
const grammar = require('./grammar.js');
const fs = require('fs').promises;

async function main() {
    const filename = process.argv[2];
    if (!filename) {
        console.log("please provide file name");
        return;
    }
    const code = (await fs.readFile(filename)).toString();
    const parser = new nearley.Parser(nearley.Grammar.fromCompiled(grammar));

    parser.feed(code);
    console.log(parser.results);
}

main().catch(err => console.log(err.stack));

upon running the parser with node parse.js example-assignment.yaklr I get this error

Error: Syntax error at line 1 col 1:

  n
  ^
Unexpected indentifier token: "n". Instead, I was expecting to see one of the following:

A indendifier token based on:
    assignment →  ● %indendifier %assignment_op literal

    at Parser.feed (path\node_modules\nearley\lib\nearley.js:343:27)
    at main (path\parse.js:14:12)

if anything is unclear about the scripts I will gladly update the information provided

NovaAppsInc commented 1 year ago

I figured it out... of course it was a typo lol instead of having identifier as identifier in moo I had it as indentifier