lexer design for performance

[jojo05]

Hi, I am really looking forward to using your library.

I suggest using a a traditional lexer (w/o goroutines) now that the library is young so that is eventually production ready (i.e. able to get best performance).

See references:

• A look at go scanner packages
• google groups discussion

That talk was about a lexer, but the deeper purpose was to demonstrate how concurrency can make programs nice even without obvious parallelism in the problem. And like many such uses of concurrency, the code is pretty but not necessarily fast.

I think it's a fine approach to a lexer if you don't care about performance. It is significantly slower than some other approaches but is very easy to adapt. I used it in ivy, for example, but just so you know, I'm probably going to replace the one in ivy with a more traditional model to avoid some issues with the lexer accessing global state. You don't care about that for your application, I'm sure.

So: It's pretty and nice to work on, but you'd probably not choose that approach for a production compiler.

-rob

[jojo05]

Apologize for the bad formatting. I will figure out markdown syntax next time

[chewxy]

Fixed the syntax for you.

I'm not entirely sure what point you're trying to make btw

I see your point. My plate's pretty full right now, and I'd appreciate a PR.

Some important key points:

• Each component needs to be able to work on itself
• Each component needs to be able to be put into a pipeline (as per the example)

[chewxy]

(updated comments)