no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
814 stars 65 forks source link

Question: implementing read macros #167

Open jasonsbarr opened 2 years ago

jasonsbarr commented 2 years ago

I'm looking to generate a lexer and parser for a Lisp-like language. Generally this should be a pretty simple task, since Lisp is really easy to parse, but there's one thing I'm not sure about.

It's common in Lisps to have read macros that transform the input string before it's tokenized. The simplest and probably most-used example in practice is something like this:

'hello

which is transformed by the reader into

(quote hello)

before being parsed.

Is there any way to do that sort of processing with a Moo lexer, or do I need to do that transformation prior to lexing?

Doing the transformation first should be a trivial exercise, but I'd like to preserve line and column info from the original code, which makes it a bit more sticky.