Tokenizer broken for x<int

[GoogleCodeExporter]

mysql> call _get_sql_tokens('p<10000');
+----+-------+-------+-------+-------+
| id | start | level | token | state |
+----+-------+-------+-------+-------+
|  1 |     2 |     0 | p     | alpha |
|  2 |     8 |     0 | 0     | error |
|  3 |     8 |     0 |       | start |
+----+-------+-------+-------+-------+

this needs to be fixed in order to really fix 
http://code.google.com/p/common-schema/issues/detail?id=14

Original issue reported on code.google.com by roland.bouman on 16 Sep 2011 at 6:03

[GoogleCodeExporter]

Fixed in r94. 

Original comment by roland.bouman on 16 Sep 2011 at 7:42

• Changed state: Fixed

[GoogleCodeExporter]

wonderful!

Original comment by shlomi.n...@gmail.com on 16 Sep 2011 at 5:00

[GoogleCodeExporter]

I expect we'll see more of these things. This is the kind of thing that would 
benefit from unit testing. I'm not really an expert on settng that kind of 
thing up though. Oh well.

Original comment by roland.bouman on 16 Sep 2011 at 5:37