blynn / nex

Lexer for Go
http://cs.stanford.edu/~blynn/nex/
GNU General Public License v3.0
416 stars 47 forks source link

Lexer doesn't return a terminal newline token? #56

Open danswartzendruber opened 3 years ago

danswartzendruber commented 3 years ago

So I was trying to migrate a BISON/FLEX basic parser written in "C" to GO. Settled on goyacc and nex. But was seeing bizarre behavior, where lines were not being processed until I entered a newline again. To eliminate questions about my grammar, I tried building the rp test program, and saw the same behavior. e.g. I enter '111 111 * 3 /', and nothing is printed until I enter another newline (or ctrl-d), at which point it prints the answer. I patched yyDebug to 3 to observe the grammar actions, and saw that it wasn't completing the parse, because the terminal newline was not being returned to the parser.

Output with yyDebug==3:

reduce 1 in: state-0 111 111 3 / lex NUM(57346) reduce 5 in: state-5 lex NUM(57346) reduce 5 in: state-5 lex ''(42) reduce 8 in: state-11 lex NUM(57346) reduce 5 in: state-5 lex '/'(47) reduce 9 in: state-12

(notice how it hasn't printed anything yet?) Now I enter another newline:

lex '\n'(10) reduce 4 in: state-6 4107 reduce 2 in: state-2

and it finishes up.

p.s. I cloned this straight out of github...