search

lark-parser / lark

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
MIT License
4.75k stars 401 forks source link

Populate the `Token.end_*` fields for ignored tokens #1309

Closed jtbraun closed 1 year ago

jtbraun commented 1 year ago

Lexer callbacks can make use of the Token.end_* fields too, even for Ignored tokens. Always populate them.

This preserves the behavior where Token objects are only created if they are used.

erezsh commented 1 year ago

Thank you for contributing!