osa1 / lexgen

A fully-featured lexer generator, implemented as a proc macro
MIT License
63 stars 7 forks source link

Implement [^'0'-'9'] syntax #57

Closed seanyoung closed 1 year ago

seanyoung commented 1 year ago

We can already do this today with #, no? This should be the same as your test lexer:

lexer! {
    Lexer -> usize;

    $ = 1,
    'a' = 2,
    _ # ['b'-'z'] = 3,
    _ = 4,
}

Am I missing anything?

Yes, you are right that it can be done with the existing syntax. It's just convenience syntax, when I first looked at lexgen I thought, why can't I do [^...]? This is how this is usually expressed, and not via _ # .... The resulting DFA will be exactly the same of course.

osa1 commented 1 year ago

Adding new syntax is often quite costly. I wrote about this a while ago on my blog: https://osa1.net/posts/2020-01-22-no-small-syntax-extensions.html it's in a different context but most of the points apply here. In this particular case, I don't think this syntax will have much benefit given that _ # ... syntax not too much more verbose and it does the same thing. So it's extra complexity (for lexgen users and developers) without much benefit.

Do you find the lack of this syntax to be a problem in practice?

seanyoung commented 1 year ago

Do you find the lack of this syntax to be a problem in practice?

Not really. I was just toying with lexgen.