Open macobo opened 9 years ago
For the tokenization process, this class seems like an useful reference: https://github.com/wvanbergen/sql_tree/blob/master/lib/sql_tree/tokenizer.rb
String tockenizing is a bit tricky in any case: http://stackoverflow.com/questions/19812597/postgresql-string-escaping-settings.
https://github.com/andialbrecht/sqlparse/blob/master/sqlparse/lexer.py#L194 does use a regex for this though.
More info at http://www.postgresql.org/docs/8.2/static/sql-syntax-lexical.html
Tricky bit is handling escape codes, this will complicate the query tokenizing logic quite a bit.