ilammy
commented
7 years ago I looked a bit around for Rust lexer/parser generators:
The first three are plain awful not as good as I have expected:
- require either nightly Rust or codegen via syntex
- have loads of transitive dependencies
- codegen = weird build, no debug
nom seems to be a reasonable one, not having any of the disadvantages above.
There is always a backup plan of using regexps directly, but we all know a joke about solving problems with regular expressions.
[ ] Prepare benchmarking code.
I would like to use some large codebase of Scheme code to have a 'typical' benchmark. For example, Chibi Scheme or whatever. Also it would be great to have automatic tracking of benchmarking results.
[ ] Profile the scanner for bottlenecks.
I have no idea on what should be used for Rust. Valgrind?
[ ] Optimize the scanner.
The general idea is to kill two birds with one stone: write a generated ASCII-only scanner using some lexer generator and couple it the existing handwritten parser. The generated parser should be used for fast scanning of ASCII-only correct code, and the handwritted parser will be used as a fallback for non-ASCII texts and invalid source code. This would give us performance boost and a formal model in one go. Of course, obvious pessimizations in the handwritten parser should be fixed as well.