Open madlep opened 1 year ago
I 100% had the intermediary step with variable assignment (returning {token, rest}
) because I knew it was going to be reviewed on-stream and I wanted it to be easier to read and focus on the sweetness of binary pattern-matching.
They did the on-stream review, so cool with optimizing it.
I think your work and review matters, but I’m no collaborator either
I 100% had the intermediary step with variable assignment (returning
{token, rest}
) because I knew it was going to be reviewed on-stream and I wanted it to be easier to read and focus on the sweetness of binary pattern-matching.They did the on-stream review, so cool with optimizing it.
Yeah, I watched that, it is was good to follow along with. The previous version is much more readable than this version. It's definitely sacrificing readability for performance here.
Was thinking about this, and worked in a bunch of optimisations to make things ✨ blazingly faster ✨
TL;DR - 3x faster, 3x less memory usage
lex/2
, and callingtokenize/2
to return a tuple of{token, rest}
, just do everything pure tail recursively. This is the main speed and memory boost because:call_last
instruction rather thancall
- just is faster.case
expression rather than separate function heads. They both work identically (and have verified the generated BEAM byte code is the same in both cases), but because we're passingtokens
, and doing another function call in each case, it gets messier as function heads. Becausetokens
is already an argument, we can just pass that along rather than having to explicitly declare it as an argument each time.Monkey.OldLexer
Apologies to @ryanwinchester . Was gonna make just a couple of tweaks, but ended up touching pretty much all the original code. The same logic is still there. It's just all being called a bit differently.