blenderskool / vyaakaran

📜 Visualize formal languages and automata
https://vyaakaran.now.sh
MIT License
35 stars 5 forks source link

Improve tokenization #1

Closed blenderskool closed 3 years ago

blenderskool commented 3 years ago

Tokenization currently doesn't use look-forward to create tokens. It groups tokens based on their individual classification. This causes - symbol to be treated as a Separator even though it could be Literal (as there is no special meaning with just - symbol in the syntax apart from its usage in -> symbol).