rrevenantt / antlr4rust

ANTLR4 parser generator runtime for Rust programming laguage
Other
404 stars 70 forks source link

Auto-substitution of TokenSource methods ? #4

Closed somidad closed 4 years ago

somidad commented 4 years ago

For example, I have the following rule

LINE_COMMENT
    : {getCharPositionInLine() == 0}? (' ' | '\t')*? '--' ~('\n'|'\r')* '\r'? '\n' ->skip
    ;

It would generate the following lexer in Rust

impl ASN_3gppLexerActions{

  fn LINE_COMMENT_sempred(_localctx: &LexerContext, pred_index:isize,
            recog:&mut <Self as Actions>::Recog
    ) -> bool {
    match pred_index {
        0=>{
          getCharPositionInLine() == 0
        }
      _ => true
    }
  }
}

Which results in a compilation error since Rust lexer does not have getCharPositionInLine(). Instaed, it has get_char_position_in_line().

I think it would be better if Rust runtime automatically substitutes Java-specific TokenSource methods including getCharPositioninInLine() to recog.get_char_position_in_line()

rrevenantt commented 4 years ago

Currently all targets will just insert raw text of lexer actions directly into generated lexer code, so it is not in scope of the rust target right now. ANTLR can add syntax extension to lexer actions similar to parser actions, but that should be decided in main ANTLR repo for all targets.

Personally, I am not sure if it is a good idea because it will work only in very trivial cases, anything more complex and you will need to write target specific code anyway.

somidad commented 4 years ago

I see. Thanks