search

bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
92 stars 6 forks source link

Specify case insensitive matching of token upon definition #12

Open ktsivkov opened 6 months ago

ktsivkov commented 6 months ago

First I want to say that I really like the api of this library, it is very easy to use.

What I want to propose:

Would be great if we could specify specific tokens to be matched case insensitive. Lets say I have a token called WHERE, would be great if it also matched where. i.e. Tokenizer.DefineTokens(TOKEN_TYPE, []string{"WHERE"}, true) or Tokenizer.DefineInsensitiveTokens(TOKEN_TYPE, []string{"WHERE"})

bzick commented 5 months ago

I think i's good idea. I'm looking at how to implement the idea