Closed gsnedders closed 3 years ago
@gsnedders Thank you for your patch. I'll review it this week.
Sorry my 4-years long reply. It was intentional that make_tokenizer()
returns a generator of tokens. It means you can do lexical analysis lazily (currently useless in the context of further parsing, but potentially beneficial) In practical terms though most uses of make_tokenizer()
require skipping whitespace tokens. Usually it mean applying a list comprehension expression, that tokenises the string eagerly:
tokenize = make_tokenizer(specs)
tokens = [t for t in tokenize(text) if t.type != "whitespace"]
Fixes #44.