blynn / nex

Lexer for Go
http://cs.stanford.edu/~blynn/nex/
GNU General Public License v3.0
416 stars 47 forks source link

nested expression question #25

Open soforth opened 9 years ago

soforth commented 9 years ago

By using nex to parse SQL, I encounter the following problem: Say two statements "BETWEEN expr AND expr" and "IF( expr AND expr)", there are two "AND" token that are different. In flex we can write rule like this:

%s BTWMODE

AND { BEGIN INITIAL; return AND; } AND { return ANDOP; } BETWEEN { BEGIN BTWMODE; return BETWEEN; } %s represents an inclusive-mode lexing.(%x exclusive-mode correspondingly) What is the equivalent counterpart of it? In addition, flex has an option "case-insensitive" to ignore case, seems nex has no such switch, so I have to write rules like /[Ss][Ee][Ll][Ee][Cc][Tt]/ to represent the "select" keyword. Is there a recommend way to achieve this?
blynn commented 9 years ago

I'm afraid nex has no start conditions. I wonder if Nex's nested nature would allow something like

/BETWEEN.*AND/ < /expr/

but even then you'd have to rewrite the expr regexes all over again, because nex doesn't (yet) support definitions.

There's also no easy way to ignore case in Nex yet.

I'd like to address these shortcomings one day, but it's not high on my to-do list. I guess at the very least I should clean up the code a little to give others a chance of patching it.