tokenizer - Githubissues

degory / ghul

compiler for the ghūl programming language

https://ghul.dev

GNU Affero General Public License v3.0

4 stars 0 forks source link

tokenizer #1

Closed degory closed 6 years ago

degory commented 7 years ago

Need to be able to transform an input stream from bytes/characters into tokens

Must integrate with Bison generated parser (expose yylex compatible interface)
Must work with Bison grammar to disambiguate potentially ambiguous input (generic brackets)

degory commented 7 years ago

Dropped Bison in favour of hand-rolled recursive descent. Don't believe parsing generic brackets needs special handling if using Pascal style definitions where we always know in advance from context if we need to parse a type