no-context / moo

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
BSD 3-Clause "New" or "Revised" License
821 stars 65 forks source link

Possible bug in applying transform functions? #74

Closed spurcell closed 6 years ago

spurcell commented 6 years ago
const moo = require('./moo')
const assert = require('assert');

// TEST: match string literals and strip quote chars in transform function
let lexer = moo.compile({
    string:       { match: /"(?:\\[btnr"\\]|[^\n"`\\])*?"/, value: text => text.slice(1, -1) },
});

// demonstrate String.slice works as we'd expect in this case
assert.equal('""'.slice(1, -1), '');

// demonstrate correct lexer behavior - quote chars removed from non-empty string match
lexer.reset('"a"');
assert.equal(lexer.next().value, 'a');

// demonstrate incorrect lexer behavior - quote chars not removed from empty string match
lexer.reset('""');
assert.equal(lexer.next().value, '');
tjvr commented 6 years ago

Oops, you're right: if the result of a transform is falseish, then Moo will ignore it! I'll fix this today. :-)

tjvr commented 6 years ago

Fixed in v0.4.3. :-)

spurcell commented 6 years ago

Much appreciated; I've deleted my workarounds.

tjvr commented 6 years ago

You're welcome! Thanks for reporting this. :-)