search

vlasovskikh / funcparserlib

Recursive descent parsing library for Python based on functional combinators
https://funcparserlib.pirx.ru
MIT License
338 stars 38 forks source link

Fix #44: make make_tokenizer return a list as parsers expect sequences #45

Closed gsnedders closed 3 years ago

gsnedders commented 8 years ago

Fixes #44.

vlasovskikh commented 8 years ago

@gsnedders Thank you for your patch. I'll review it this week.

vlasovskikh commented 3 years ago

Sorry my 4-years long reply. It was intentional that make_tokenizer() returns a generator of tokens. It means you can do lexical analysis lazily (currently useless in the context of further parsing, but potentially beneficial) In practical terms though most uses of make_tokenizer() require skipping whitespace tokens. Usually it mean applying a list comprehension expression, that tokenises the string eagerly:

tokenize = make_tokenizer(specs)
tokens = [t for t in tokenize(text) if t.type != "whitespace"]