no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
821 stars 65 forks source link

Case insensitive keywords? #46

Closed mx-scissortail closed 7 years ago

mx-scissortail commented 7 years ago

I've got a use case where the keywords in my language are case insensitive but other things (e.g. the contents of strings) aren't. Is there any way I can make a lexer that recognizes these? I tried keyword: [/foo/i, ...] but it complains that the "/i" flag isn't allowed.

tjvr commented 7 years ago

Right. Moo combines all the tokens into a single RegExp, so supporting both case-sensitive and insensitive tokens is difficult. Can you tell me more about the language? That might help :-)

One suggestion would be to do e.g. /[fF][oO][oO]/. You could write a function to generate such RegExps from strings, you needn't do it by hand. :-)

mx-scissortail commented 7 years ago

Ah, yeah. It's a pretty simple DSL where each line starts with a case insensitive keyword, e.g.:

enter: (some-terrible-lisp-expression). ;comment
Enter: (another-terrible-lisp-expression).
delete: "string".

etc.

I think generating the individual cases won't be too hard, so that will probably work. Thanks.

tjvr commented 7 years ago

Glad I could help! :-)

Be aware that you might run into an edge case if you have identifiers too: https://github.com/tjvr/moo#keywords —moo only handles that if your keywords are specified as strings.

deltaidea commented 7 years ago

Maybe it would be a good idea to add something like this:

moo.compile({
  evenOdd: { caseInsensitive: true, keyword: ['even', 'odd'] }
})