Closed wduquette closed 4 years ago
This pull request adds a new type, Tokenizer, which allows a string slice to be parsed into slices while still supporting "peek()", and uses it in the list-syntax parser. Tokenizer will eventually replace CharPtr and EvalPtr.
This pull request adds a new type, Tokenizer, which allows a string slice to be parsed into slices while still supporting "peek()", and uses it in the list-syntax parser. Tokenizer will eventually replace CharPtr and EvalPtr.