help "customizing" the lexer

[danielo515]

Hello,

I just want to make some experiments with livescript. Basically I want to alias function to another shorther keyword. So I want to be able to use

def sum a, b
    a + b

instead of

function sum a, b
    a + b

At first I though it should be some simple change, go to the lexer and just add another token for function... but seems that livescript token recognition is very complex. I read several times the lexer and the grammar files. I more or less understand how do they work, but It's been very hard to me to understand where certain keywords are defined, function being one of them.

If you could help me with this task I'll be very grateful.

Regards

[vendethiel]

The easiest is probably to support both. Pretend a def is actually a function. Also augment the LS_KEYWORDS variable.

[rhendric]

Look at do-ID in the lexer; that's where keywords are recognized. If you add a case for 'def' and set tag = \FUNCTION; id = '', that'll probably get you most of the way there. Also look at LS_KEYWORDS as ven suggested.

[danielo515]

Hello @vendethiel and @rhendric , What you say is more or less what I tried, but it was obviously wrong.

This is what I did: File: lexer.ls line: 167 (this is on do-ID) I added:

        case 'catch' 'function' 'def'
            id = ''

Obviously that was not enough, so I also tried (this way directly on the compiled file): File: lexer.js, line: 656 (this is under doLiteral function)

  case 'def':
    tag = 'FUNCTION'
    break;

[danielo515]

Hello again @rhendric and @vendethiel

Following your suggestions here is what I have changed: https://github.com/gkz/LiveScript/compare/master...danielo515:feature/defFunctionShorthand

Now it works as expected, however the repl does not understand how to handle a function defined with the new def keyword. I get an error about an unexpected newline:

{ Error: Parse error on line 1: Unexpected 'NEWLINE'
    at Parser.parseError (/Users/danielo/programmingTest/LiveScript/lib/parser.js:787:21)
    at Parser.parse (/Users/danielo/programmingTest/LiveScript/lib/parser.js:854:22)
    at Object.exports.compile (/Users/danielo/programmingTest/LiveScript/lib/index.js:51:20)
    at Interface.<anonymous> (/Users/danielo/programmingTest/LiveScript/lib/repl.js:240:43)
    at Interface.emit (events.js:182:13)
    at Interface.EventEmitter.emit (domain.js:442:20)
    at Interface._onLine (readline.js:290:10)
    at Interface._line (readline.js:638:8)
    at Interface._ttyWrite (readline.js:919:14)
    at Interface._ttyWrite (/Users/danielo/programmingTest/LiveScript/lib/repl.js:77:22)
  hash:
   { text: undefined,
     token: 'NEWLINE',
     line: 0,
     loc:
      { first_line: 0, first_column: 10, last_line: 0, last_column: 10 },
     expected: [ '\'INDENT\'' ] } }

Should I update the repl.ls too ?

[rhendric]

Ah yeah, the REPL's handling of multiline constructs is not elegant at all. If you want that to work, you'll need to modify some regexes in there.

By the way, any time the REPL prematurely interprets a line break as the end of a read, you can instead use Ctrl-J to introduce a line break that doesn't end the read. This is obviously not what you'd want to rely upon to fix this issue for your experiment, but it may help you while debugging.

[danielo515]

Thanks again @rhendric , For what I want what I have is good enough. On the repl I can just use the function keyword. I don't think nobody wants this to end on the master branch (but it will be cool 😄 ) so it's good enough for me to work with it. If at any time, someone wants to integrate this into the official version I'll ve happy to update the repl.

[rhendric]

Cheers then!

(You're right, that alias would be unlikely to find its way into this version of LiveScript, but I do think that the ability to define keyword aliases like this without modifying the compiler would be a nifty feature in, say, a hypothetical successor language that someone might be working on in secret.)