If Scanner returns a token, there is only one state where it cannot simply be reset with no resulting subsequent token after the caller is done with the token: pi_maybe_end. It would be a great simplification opportunity if this could be omitted, and the scanner could just be cleanly reset before starting on the next token in TokenReader.
This is also a reminder to me to double-check on the similar CDATA ending states, which look like they may be handled incorrectly, but I need to confirm (and add a comment explaining why the current behavior is correct if it turns out to be right).
If
Scanner
returns a token, there is only one state where it cannot simply be reset with no resulting subsequent token after the caller is done with the token:pi_maybe_end
. It would be a great simplification opportunity if this could be omitted, and the scanner could just be cleanly reset before starting on the next token inTokenReader
.This is also a reminder to me to double-check on the similar CDATA ending states, which look like they may be handled incorrectly, but I need to confirm (and add a comment explaining why the current behavior is correct if it turns out to be right).