no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
821 stars 65 forks source link

Add keywordsCaseInsensitive option #78

Closed camullen closed 5 years ago

camullen commented 6 years ago

I added an option to use case insensitive keywords, but by default they are case sensitive, and the fast string lookup function is unaffected. Essentially, when keywordsCaseInsensitive is true, the keywords themselves are converted to upper case when creating the fast string lookup function, and the function converts values it is searching for to upper case before matching.

The option works as follows:

   let lexer = compile({
      identifier: {
        match: /[a-zA-Z]+/,
        keywordsCaseInsensitive: true,
        keywords: {
          'kw-class': 'class',
          'kw-def': 'def',
          'kw-if': 'if',
        },
      },
      space: {match: /\s+/, lineBreaks: true},
    })
duartealexf commented 6 years ago

Oh I really need this... is anyone considering it? ... @tjvr ?

tjvr commented 5 years ago

Thanks for your contribution!

I'm going to close this in favour of the general-purpose type transform added in #85.

I need to document it better, but you can now do something like this:


const caseInsensitiveKeywords = defs => {
  const keywords = moo.keywords(defs)
  return value => keywords(value.toLower())
}

let lexer = compile({
  identifier: {
    match: /[a-zA-Z]+/,
    type: caseInsensitiveKeywords({
      'kw-class': 'class',
      'kw-def': 'def',
      'kw-if': 'if',
    }),
  },
  space: {match: /\s+/, lineBreaks: true},
})