Recursive Descent parsers tend to lex the document as they go, rather than all up front. This allows you to know what to do in the case a bad token is lexed in the context of the current parsing state.
The existing lexer (elixir_tokenizer) is designed to work with a parser generator (yecc), so it parses it all up front, and if it reaches a bad token, it bails and returns an error.
Solution
Refactor the existing :elixir_tokenizer (vendored as :spitfire_tokenzier to enable on demand lexing.
The API of the module should basically consist of
new - creates a new lexer state instance from a source code string
next_token - returns the next token in the document and the new lexer state
The token structure should stay the same and contain the same semantics.
Considerations
the existing implementation has a special interpolation module
conversion from Erlang to Elixir is not necessary, but might eventually be done
Description
Recursive Descent parsers tend to lex the document as they go, rather than all up front. This allows you to know what to do in the case a bad token is lexed in the context of the current parsing state.
The existing lexer (elixir_tokenizer) is designed to work with a parser generator (yecc), so it parses it all up front, and if it reaches a bad token, it bails and returns an error.
Solution
Refactor the existing
:elixir_tokenizer(vendored as:spitfire_tokenzierto enable on demand lexing.The API of the module should basically consist of
new- creates a new lexer state instance from a source code stringnext_token- returns the next token in the document and the new lexer stateThe token structure should stay the same and contain the same semantics.
Considerations