The FETCH comand (#2) has optional tokens. One way to support optional tokens is to have a puchBack() function in the parser that retries a token.
At the moment, a pushBack() parsing function is not supported by the lexer because it uses flags to control the behaviour of the next() function. The flag may change when a token is retried.
Both the parser and lexer need to support a pushBack().
Also, the current method of passing flags to lexer.next() is a hack to support data dependence e.g a bare string is a tag or an astring depending on the position. The flag should be replaced by having different entry points e.g:
The FETCH comand (#2) has optional tokens. One way to support optional tokens is to have a puchBack() function in the parser that retries a token.
At the moment, a pushBack() parsing function is not supported by the lexer because it uses flags to control the behaviour of the next() function. The flag may change when a token is retried.
Both the parser and lexer need to support a pushBack().
Also, the current method of passing flags to lexer.next() is a hack to support data dependence e.g a bare string is a tag or an astring depending on the position. The flag should be replaced by having different entry points e.g: