puffnfresh / roy

Small functional language that compiles to JavaScript.
http://roy.brianmckenna.org/
MIT License
834 stars 74 forks source link

Lexer refactor #188

Closed rtfeldman closed 10 years ago

rtfeldman commented 10 years ago

Refactored lexer in support of removing let (which is almost there; 1 failing test cases remaining due to where).

The main changes are:

  1. Put the keywords in a map, so we can easily check if something is a keyword or not by checking its presence in the map.
  2. Parameterize the tokenize functions on chunk, so we can process a substring of chunk instead of the whole thing.
  3. Pull out a tokenise function that can be used outside the main exports.tokenise.

This is a big enough change to a critical piece of code that I figured a PR would be appropriate.

rtfeldman commented 10 years ago

Any objections to merging this in?

joneshf commented 10 years ago

I'm game.