LEMON parser generator modified to generate Rust code.

Lemon source and SQLite3 grammar were last synced as of July 2024.

Unsupported

Unsupported Grammar syntax

• %token_destructor: Code to execute to destroy token data
• %default_destructor: Code for the default non-terminal destructor
• %destructor: Code which executes whenever this symbol is popped from the stack during error processing

https://www.codeproject.com/Articles/1056460/Generating-a-High-Speed-Parser-Part-Lemon https://www.sqlite.org/lemon.html

SQLite

SQLite lexer and SQLite parser have been ported from C to Rust. The parser generates an AST.

Lexer/Parser:

• Keep track of position (line, column).
• Streamable (stop at the end of statement).
• Resumable (restart after the end of statement).

Lexer and parser have been tested with the following scripts:

• https://github.com/bkiers/sqlite-parser/tree/master/src/test/resources
• https://github.com/codeschool/sqlite-parser/tree/master/test/sql/official-suite which can be updated with script in https://github.com/codeschool/sqlite-parser/tree/master/test/misc

TODO:

• [ ] Check generated AST (reparse/reinject)
• [ ] If a keyword in double quotes is used in a context where it cannot be resolved to an identifier but where a string literal is allowed, then the token is understood to be a string literal instead of an identifier.
• [ ] Tests
• [ ] Do not panic while parsing
• [x] CREATE VIRTUAL TABLE args
• [ ] Zero copy (at least tokens)

Unsupported by Rust

• #line directive

API change

• No ParseAlloc/ParseFree anymore

Features not tested

• NDEBUG
• YYNOERRORRECOVERY
• YYERRORSYMBOL

To be fixed

• RHS are moved. Maybe it is not a problem if they are always used once. Just add a check in lemon...
• %extra_argument is not supported.
• Terminal symbols generated by lemon should be dumped in a specified file.

Raison d'être

• lemon_rust does the same thing but with an old version of lemon. And it seems not possible to use yystack as a stack because items may be access randomly and the top+1 item can be used.

• lalrpop would be the perfect alternative but it does not support fallback/streaming (see this issue) and compilation/generation is slow.