Closed GoogleCodeExporter closed 9 years ago
What steps will reproduce the problem?
the last token is missing every time
What is the expected output? What do you see instead?
Token list returned by the Tokenizer is always missing the last token. It seems
that
it stores the last token as PREFIX, but forgets to return it.
Original comment by lemontree.cool
on 10 May 2008 at 2:19
Original comment by lemontree.cool
on 21 May 2009 at 2:43
Original comment by lemontree.cool
on 23 Jul 2009 at 3:37
Original issue reported on code.google.com by
lemontree.cool
on 10 May 2008 at 10:52