antlr / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
BSD 3-Clause "New" or "Revised" License
17.04k stars 3.27k forks source link

JavaScript runtime fails to parse input that can be parsed by `grun` with same grammar #4619

Open KerimG opened 4 months ago

KerimG commented 4 months ago

Hi all, this is the first time I'm using ANTLR4 with a JavaScript target I tried testing the grammar in the ANTLR4 lab and got the following additional information. It seems to me, as though, there's a problem with the token recognition in the JavaScript runtime.

image

replacing * with foo returns this:

image

in grun everything looks like I'd expect:

image

with the following output:

[@0,0:5='SELECT',<'SELECT'>,1:0]
[@1,6:8='\n  ',<WS>,channel=1,1:6]
[@2,9:9='*',<'*'>,2:2]
[@3,10:10='\n',<WS>,channel=1,2:3]
[@4,11:14='FROM',<'FROM'>,3:0]
[@5,15:17='\n  ',<WS>,channel=1,3:4]
[@6,18:25='EMPLOYEE',<ID_LITERAL>,4:2]
[@7,26:26='\n',<WS>,channel=1,4:10]
[@8,27:26='<EOF>',<EOF>,5:0]
(root (subselect (selectClause SELECT (selectList (selectItem *))) (fromClause FROM (tableName EMPLOYEE))) <EOF>)

Btw, is there -token equivalent in JavaScript?

Grammar:

grammar Sql;

options {
    caseInsensitive = true;
}

root: subselect (EOF | ';' EOF);

subselect 
    : selectClause fromClause;

selectClause 
    : 'SELECT' ( 'ALL' | 'DISTINCT' )? selectList
    ;

selectList
    : selectItem ( ',' selectItem )*
    ;

selectItem
    : ASTERISK
    | columnName
    ;

fromClause 
    : 'FROM' tableName
    ;

columnName: ID_LITERAL;

viewName: ID_LITERAL;

tableName: ID_LITERAL;

ASTERISK : '*';

WS
    : [ \t\r\n]+ -> channel(HIDDEN)
    ;

ID_LITERAL: IDENTIFIER | IDENTIFIER'.'IDENTIFIER;

fragment IDENTIFIER: '_'? [A-Z_0-9]+;
kaby76 commented 4 months ago

Your grammar, "Sql", is a combined grammar. Did you clear out the "Lexer grammar" tab in Antlr Lab? Antlr Lab sets up the UI with a default example, which is a split grammar.

Antlr Lab is down, again. It would be good to add to "cron" a periodic reboot. No "code" of the server needs to change, so it would be an easy fix for something that keeps crashing.

Antlr Lab does not run a JavaScript target of the parser. It runs an "interpreter" in a Java process.