Allow using an alternative tokenizer

we-like-parsers / pegen

PEG parser generator for Python

https://we-like-parsers.github.io/pegen/

MIT License

150 stars 32 forks source link

Allow using an alternative tokenizer #65

Open MatthieuDartiailh opened 2 years ago

MatthieuDartiailh commented 2 years ago

I am not sure this approach is the nicest and it looks a bit brittle in places but it would allow to use alternate tokenizers with a well defined interface which would be nice.

Comments welcome !

pablogsal commented 2 years ago

This needs rebasing 😅

MatthieuDartiailh commented 2 years ago

Rebased but will #71 to pass on 3.10

MatthieuDartiailh commented 2 years ago

Rebased again so that CIs should pass @pablogsal once the 3.11beta5 is live could you review this ?

MatthieuDartiailh commented 1 year ago

@pablogsal I resolved the conflict and hopefully fixed the 3.8 failure. Could you please review ?

0dminnimda commented 5 months ago

It would be nice to finally merge this