search

ianh / owl

A parser generator for visibly pushdown languages.
MIT License
746 stars 21 forks source link

How would one handle a macro/define keyword, is it possible currently? #18

Closed warvstar closed 4 years ago

warvstar commented 4 years ago

I'm trying to implement a macro keyword. Is it possible to do something like? 'def' identifier remaining_line: define

If not, where would be a good place for me to start looking to add such a feature?

Also this is a really cool project! I've tested out quite a few different parser generators and this one just blows everything out of the water so far for what I'm trying to use it for.

ianh commented 4 years ago

Syntactic macros are hard for parser generators in general—even if you could match remaining_line, you'd need a way to splice those tokens into the token stream at the point that the macro is used. It's almost possible to do this splicing in Owl by returning the tokens from a custom tokenize function, but the way it is now, you can only return user-defined tokens, not built-in ones.

One way of solving this would be to add support for completely custom tokenizers. Then all the macro matching and substitution could happen within the tokenizer. The grammar wouldn't even see 'def' tokens—they'd be handled ahead of time, like in the C preprocessor.

Another would be to add macro matching and expansion to Owl itself. This would be a bigger project, but it would let you use macros in interpreter mode, which would be cool.

I'm open to suggestions for other ideas, too, or more details about what you're doing that might help find the right solution.

warvstar commented 4 years ago

I'm not sure what I was thinking, it would be trivial and make more sense to just do ahead of time with a simple string replace, as you have mentioned.

Your idea at the end would be cool for the interpreter though.

On Fri, Jan 3, 2020, 4:40 PM Ian Henderson notifications@github.com wrote:

Syntactic macros are hard for parser generators in general—even if you could match remaining_line, you'd need a way to splice those tokens into the token stream at the point that the macro is used. It's almost possible to do this splicing in Owl by returning the tokens from a custom tokenize function, but the way it is now, you can only return user-defined tokens, not built-in ones.

One way of solving this would be to add support for completely custom tokenizers. Then all the macro matching and substitution could happen within the tokenizer. The grammar wouldn't even see 'def' tokens—they'd be handled ahead of time, like in the C preprocessor.

Another would be to add macro matching and expansion to Owl itself. This would be a bigger project, but it would let you use macros in interpreter mode, which would be cool.

I'm open to suggestions for other ideas, too, or more details about what you're doing that might help find the right solution.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ianh/owl/issues/18?email_source=notifications&email_token=ACBQKL5PDS36B7DTUWJEBQLQ37EFBA5CNFSM4KCSC73KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEICKRIA#issuecomment-570730656, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBQKL2WNW72BQ4FI6KFU5LQ37EFBANCNFSM4KCSC73A .

ianh commented 4 years ago

Okay, I'll close this then. Feel free to open a new issue if there's something else you need.