Tokenizer: trailing token missing

[GoogleCodeExporter]

What steps will reproduce the problem?
1. seperate the column/lineNum logics from Tokenizer
2. remove the prefix logic

What is the expected output? What do you see instead?
Token list returned by Tokenizer is always missing the last token. Probably
it stores the last token as PREFIX, but forget to return it.

Original issue reported on code.google.com by lemontree.cool on 10 May 2008 at 10:52

[GoogleCodeExporter]

What steps will reproduce the problem?
the last token is missing every time

What is the expected output? What do you see instead?
Token list returned by the Tokenizer is always missing the last token. It seems 
that
it stores the last token as PREFIX, but forgets to return it.

Original comment by lemontree.cool on 10 May 2008 at 2:19

[GoogleCodeExporter]

Original comment by lemontree.cool on 21 May 2009 at 2:43

• Added labels: Component-lex
• Removed labels: Component-Logic

[GoogleCodeExporter]

Original comment by lemontree.cool on 23 Jul 2009 at 3:37

• Changed state: Fixed