Based on the TMLanguage, it is possible to tokenize various inputs and then compare them with a token list. This makes test_rules.txt unnecessary in practice (good for a demo at most), since unit tests can be used to ensure that there is no breakage and that tokenization always happens as expected.
Based on the TMLanguage, it is possible to tokenize various inputs and then compare them with a token list. This makes test_rules.txt unnecessary in practice (good for a demo at most), since unit tests can be used to ensure that there is no breakage and that tokenization always happens as expected.
See: https://www.npmjs.com/package/vscode-textmate