Add TokenizerImpl

[GoogleCodeExporter]

Default Tokenizer implemetation. It can accept a list of TokenType/Regex
pairs as the lexical specification. It will use the java regex
implementation to do the token matching.

A simple application of the parser and the tokenizer is for the
Rule.createRule():
lexical spec:
SYMBOL ::= "[A-Z]+"
SEPARATOR ::= "::="
SPACE ::= " *" // should be skiped

syntax spec:
RULE ::= SYMBOL SEPARATOR SYMBOL
RULE ::= RULE SYMBOL

Original issue reported on code.google.com by lemontree.cool on 2 Oct 2008 at 4:23

[GoogleCodeExporter]

add special SPACE and COMMENT *token name*. these tokens will be ignored.

Original comment by lemontree.cool on 4 Oct 2008 at 6:35

[GoogleCodeExporter]

* TokenizerImpl should accept a Map<String, Regex> as the parameter for the 
constructor.
* TokenizerImpl should be abstract
* Subclass should provide the lexcial specifications by calling 
super(Map<String,Regex>)

Original comment by lemontree.cool on 4 Oct 2008 at 6:38

[GoogleCodeExporter]

* TokenizerImpl.tokenize() should accept a Reader as parameter, and return a
List<Terminal> for the tokens and null for error.

Original comment by lemontree.cool on 4 Oct 2008 at 6:39

[GoogleCodeExporter]

* TokenizerImpl shouldn't use the constructor parameter for the lexcial
specification. Instead, it should have an abstract factory method called
getLexConfig() which will return a Map<String,Regex>.

Original comment by lemontree.cool on 4 Oct 2008 at 6:42

[GoogleCodeExporter]

Original comment by lemontree.cool on 4 Oct 2008 at 3:11

• Added labels: Priority-Medium
• Removed labels: Priority-Low

[GoogleCodeExporter]

Original comment by lemontree.cool on 21 May 2009 at 2:45

• Added labels: Component-lex

[GoogleCodeExporter]

Original comment by lemontree.cool on 18 Jul 2009 at 11:55

• Changed state: WontFix