Closed lsp-ableton closed 2 years ago
@lsp-ableton Actually you can already track source information if you use funcparserlib.lexer.make_tokenizer()
that generates an iterable of Token
objects. You can also come up with your custom tokens that do something like that. It doesn't affect the parser itself since it happens at the lexer level.
Currently, the parser state is only available when an error occurs. It can also be useful to include source information as part of a parse tree. That is, when using
rshift
to generate a node in the tree, it would be helpful to also include the source file positions which were parsed to generate the node.If I'm understanding the code correctly, it shouldn't be too difficult to include this information as part of parsing. We could use state before running a parser as the
source-start
value and the state after running the parser forsource-end
. I'd be happy to contribute this change, but I wonder if it's welcome.Basically, this is all I'm talking about:
Maybe it should be implemented another way, but this at least produces the desired behaviour for me