Closed GoogleCodeExporter closed 9 years ago
add special SPACE and COMMENT *token name*. these tokens will be ignored.
Original comment by lemontree.cool
on 4 Oct 2008 at 6:35
* TokenizerImpl should accept a Map<String, Regex> as the parameter for the
constructor.
* TokenizerImpl should be abstract
* Subclass should provide the lexcial specifications by calling
super(Map<String,Regex>)
Original comment by lemontree.cool
on 4 Oct 2008 at 6:38
* TokenizerImpl.tokenize() should accept a Reader as parameter, and return a
List<Terminal> for the tokens and null for error.
Original comment by lemontree.cool
on 4 Oct 2008 at 6:39
* TokenizerImpl shouldn't use the constructor parameter for the lexcial
specification. Instead, it should have an abstract factory method called
getLexConfig() which will return a Map<String,Regex>.
Original comment by lemontree.cool
on 4 Oct 2008 at 6:42
Original comment by lemontree.cool
on 4 Oct 2008 at 3:11
Original comment by lemontree.cool
on 21 May 2009 at 2:45
Original comment by lemontree.cool
on 18 Jul 2009 at 11:55
Original issue reported on code.google.com by
lemontree.cool
on 2 Oct 2008 at 4:23