search

rzel / mini-java

Automatically exported from code.google.com/p/mini-java
0 stars 0 forks source link

Add RegexTokenizer #35

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
RegexTokenizer should take care of escaped characters and special
characters like ".", "(", ")", etc. The simpleset version doesn't support
character set ([a-zA-Z_]).

Original issue reported on code.google.com by lemontree.cool on 2 Oct 2008 at 3:39

GoogleCodeExporter commented 9 years ago 
RegexTokenizer should expand the specail characters into alternatives,
like: \d --> (0|1|2|3...8|9).

NOTE: there would be a performance impact for this solution.

Original comment by lemontree.cool on 3 Oct 2008 at 4:07

GoogleCodeExporter commented 9 years ago

Original comment by lemontree.cool on 21 May 2009 at 2:44

GoogleCodeExporter commented 9 years ago

Original comment by lemontree.cool on 18 Jul 2009 at 11:56