Open gwttk opened 5 years ago
public Collection<Token> tokenize(final String text, final Collection<Emit> collectedEmits) {
final Collection<Token> tokens = new ArrayList<>();
int lastCollectedPosition = -1;
for (final Emit emit : collectedEmits) {
if (emit.getStart() - lastCollectedPosition > 1) {
tokens.add(createFragment(emit, text, lastCollectedPosition));
}
tokens.add(createMatch(emit, text));
lastCollectedPosition = emit.getEnd();
}
if (text.length() - lastCollectedPosition > 1) {
tokens.add(createFragment(null, text, lastCollectedPosition));
}
return tokens;
}
ans use it like this:
trie.tokenize(text, trie.parseText(text, new StatefulEmitHandler() {
@Override
public boolean emit(Emit emit) {
// TODO Auto-generated method stub
return false;
}
@Override
public List<Emit> getEmits() {
// TODO Auto-generated method stub
return null;
}
}));
hmmm the dup text
looks ugly, but I really don't know how to get rid of it.
I was looking at the
tokenize
function, and need things likeEmitHandler
.Then I checked the code of
tokenize
, and realized that it's actually a stage afterparseText
. So why not use a pattern liketokenize(parseText(...))
, so we can enjoy all the new features offered byparseText
! Right now we already have a fewparseText
variants, and there may be more in the future!