Tokenization currently doesn't use look-forward to create tokens. It groups tokens based on their individual classification.
This causes - symbol to be treated as a Separator even though it could be Literal (as there is no special meaning with just - symbol in the syntax apart from its usage in -> symbol).
Tokenization currently doesn't use look-forward to create tokens. It groups tokens based on their individual classification. This causes
-
symbol to be treated as aSeparator
even though it could beLiteral
(as there is no special meaning with just-
symbol in the syntax apart from its usage in->
symbol).