More flexible tokenize() function

[gwttk]

I was looking at the tokenize function, and need things like EmitHandler.
Then I checked the code of tokenize, and realized that it's actually a stage after parseText. So why not use a pattern like tokenize(parseText(...)), so we can enjoy all the new features offered by parseText! Right now we already have a few parseText variants, and there may be more in the future!

[gwttk]

    public Collection<Token> tokenize(final String text, final Collection<Emit> collectedEmits) {
        final Collection<Token> tokens = new ArrayList<>();
        int lastCollectedPosition = -1;

        for (final Emit emit : collectedEmits) {
            if (emit.getStart() - lastCollectedPosition > 1) {
                tokens.add(createFragment(emit, text, lastCollectedPosition));
            }

            tokens.add(createMatch(emit, text));
            lastCollectedPosition = emit.getEnd();
        }

        if (text.length() - lastCollectedPosition > 1) {
            tokens.add(createFragment(null, text, lastCollectedPosition));
        }

        return tokens;
    }

ans use it like this:

trie.tokenize(text, trie.parseText(text, new StatefulEmitHandler() {

    @Override
    public boolean emit(Emit emit) {
        // TODO Auto-generated method stub
        return false;
    }

    @Override
    public List<Emit> getEmits() {
        // TODO Auto-generated method stub
        return null;
    }
}));

hmmm the dup text looks ugly, but I really don't know how to get rid of it.