Explicit ordering (prioritize the tokens in the order they appear in the grammar)
Single DFA construction (combining every token kind into a single DFA)
Single DFA approach is the best, but the most complicated one. I guess the implementation can start with the longest match approach is it's relatively easy, and then to DFA construction in the future.
The current implementation is very basic and even prone to unexpected behavior! Instead one of the following could be used to replace it:
Single DFA approach is the best, but the most complicated one. I guess the implementation can start with the longest match approach is it's relatively easy, and then to DFA construction in the future.