While finding the current token, WordTokenizer would run through all MentionSpans in the text to determine the span index closest to the cursor on either side. This is rather wasteful and, in the case of getSearchEndIndex(), even unnecessary.
The idea is to use nextSpanTransition() to iterate sequentially till the cursor over batches of spans to get the closest last span end index in getSearchStartIndex.
For getSearchEndIndex, nextSpanTransition() does exactly what it needs without ever looking at any MentionSpans.
While finding the current token, WordTokenizer would run through all MentionSpans in the text to determine the span index closest to the cursor on either side. This is rather wasteful and, in the case of getSearchEndIndex(), even unnecessary.
The idea is to use nextSpanTransition() to iterate sequentially till the cursor over batches of spans to get the closest last span end index in getSearchStartIndex.
For getSearchEndIndex, nextSpanTransition() does exactly what it needs without ever looking at any MentionSpans.