search

shnewto / bnf

Parse BNF grammar definitions
MIT License
258 stars 22 forks source link

lalrpop exploration #35

Closed CrockAgile closed 4 years ago

CrockAgile commented 6 years ago

I just read thru the lalrpop documentation and it seems worth investigating the readability & performance comparison to the current nom usage. This exploration task could just be done on its own branch and never merged and it would serve its purpose. But if it stumbles into some improvements, then it could be up for consideration.

shnewto commented 6 years ago

@CrockAgile I'll plan on taking some time to read the lalrpop docs. I'm definitely interested potential for value it would add to this project. Keep me posted on any details.

shnewto commented 4 years ago

Just remembered this issue and something I'd bumped into trying to work with significant whitespace reading the Writing a custom lexer chapter in the lalrpop book. Looks like there's a way work around it (the point of that chapter) but it wasn't the way I'd hoped. I think it's a deal breaker for me, for now.

By default, LALRPOP generates a tokenizer that skips all whitespace -- including newlines. What we want is to capture whitespace characters and ignore the rest as comments, and LALRPOP does the opposite of that.

At the moment, LALRPOP doesn't allow you to configure the default tokenizer. In the future it will become quite flexible, but for now we have to write our own.