rrevenantt / antlr4rust

ANTLR4 parser generator runtime for Rust programming laguage
Other
398 stars 70 forks source link

Hidden channel tokens at the start of the parsed string are not ignored #66

Open dmitrii-ubskii opened 1 year ago

dmitrii-ubskii commented 1 year ago

Consider the following grammar:

grammar hidden_test;

list: VAR_ ( ',' VAR_ )* ;
VAR_: '$' [a-zA-Z0-9][a-zA-Z0-9_-]* ;

WS : [ \t\r\n]+ -> channel(HIDDEN) ;

Parsing " $abc, $def" in Java or on http://lab.antlr.org/ produces the expected parse tree. But trying to parse this string in Rust via parser.list() produces the following error instead:

line 1:0 extraneous input ' ' expecting VAR_
newca12 commented 1 year ago

An issue indeed. I guesss the culprit is here : https://github.com/rrevenantt/antlr4rust/blob/master/src/common_token_stream.rs#L23 The first token is consumed there before next_token_on_channel function can filter it on the channel criteria.