asfimport
commented
8 years ago Timothy M. Rodriguez (migrated from JIRA)
(dsmiley: edited formatting only)
Some additional information:
Missing features & possible future improvements:
Despite the offset source flexibility and accuracy options of this highlighter, it continues to be the case that some highlighters have unique features. The following features are in the standard Highlighter (and possibly FastVectorHighlighter) but are not in the UnifiedHighlighter (and thus not PostingsHighlighter either since UH is derived from PH):
- Being able to disable “requireFieldMatch” to thus highlight a query insensitive to whatever fields are mentioned in the query.
- Using boosts in the query to weight passages.
- Regex pased passage delineation. Though I’m unsure if anyone cares given the existing BreakIterator options available.
Aside from addressing the feature gaps listed above, there are a couple known things that would be nice to add:
- The phrase highlighting (implemented by PhraseHelper) could be made more accurate, and probably faster too, by using techniques in @romseygeek's Luwak system that uses the Lucene SpanCollector API introduced in Lucene 5.3. It wasn’t done this way to begin with because this highlighter was developed originally for Lucene 4.10.
- Wildcard queries usually use TokenStreamFromTermVector, which uninverts the terms out of a Terms index. Instead, we now think it would be better to create a bunch of PostingsEnum for each matching term. This would bring about some simplifications and efficiencies, and can lead to better passage relevancy. A bonus would be aggregating terms matching the same automata into a merged PostingsEnum that has a freq() based on the sum of the underlying matching terms.
Changes from the PostingsHighlighter
- The UH is more stateful
- Holds the IndexSearcher instead of asking most methods to pass it through.
- Options now have simple setters, and the per-field getters return these. This means the common case of a setting being non-specific to a field doesn’t require subclassing.
- Multi-valued field handling is improved to ensure that a passage will never span across values, plus it honors the positionIncrementGap for an analyzed offset source. See MultiValueTokenStream and SplittingBreakIterator.
- The PH caches all content to be highlighted for all docs and then highlights it all. The UH has a limit on this which led to a batching approach. But if all fields use an Analyzer or if more than one use term vectors, then instead highlighting happens one doc at a time since the up-front content caching is not helpful.
- No longer tries to re-use PostingsEnums (or TermsEnum or LeafReader) from one doc to the next. This really simplified some code; it didn’t seem worth it.
- MultiTermHighlighting’s fake PostingsEnum was made Closeable and we close it to guard against ramifications of exceptions being thrown during highlighting (e.g. a BreakIterator bug or TokenStream bug). Nasty to debug!
- (from standard Highlighter) TokenStreamFromTermVector: optimizations to uninvert filtered (thus sparse) Terms.
Non-Core Dependencies
- MemoryIndex: For Analyzer based highlighting when phrases need to be highlighted accurately.
- Standard Highlighter things:
- TokenStreamFromTermVector: For most multi-term queries. The UH actually has its own derived copy that has been optimized to handle filtered (thus sparse) Terms. With further work, we could switch to a different approach and remove it (as indicated earlier). For as long as it stays, it’s also possible to replace the existing one with this if we want to do that.
- WeightedSpanTermExtractor: For highlighting phrases accurately to re-use it’s SpanQuery conversion and rewrite detecting abilities. Perhaps these parts of WSTE could move to general SpanQuery utilities.
- TermVectorLeafReader: When highlighting offsets from term vectors.
- PostingHighlighter things:
- Technically, Nothing however it has multiple copies of some things that have not been modified: Passage, PassageScorer, PassageFormatter, DefaultPassageFormatter.
- Note: Utility BreakIterators are of use to the PH, UH, and even the FVH: WholeBreakIterator, CustomSeparatorBreakIterator. Maybe they should move to a utils package that isn’t in any of these highlighters?
The UnifiedHighlighter is an evolution of the PostingsHighlighter that is able to highlight using offsets in either postings, term vectors, or from analysis (a TokenStream). Lucene’s existing highlighters are mostly demarcated along offset source lines, whereas here it is unified – hence this proposed name. In this highlighter, the offset source strategy is separated from the core highlighting functionalty. The UnifiedHighlighter further improves on the PostingsHighlighter’s design by supporting accurate phrase highlighting using an approach similar to the standard highlighter’s WeightedSpanTermExtractor. The next major improvement is a hybrid offset source strategythat utilizes postings and “light” term vectors (i.e. just the terms) for highlighting multi-term queries (wildcards) without resorting to analysis. Phrase highlighting and wildcard highlighting can both be disabled if you’d rather highlight a little faster albeit not as accurately reflecting the query. We’ve benchmarked an earlier version of this highlighter comparing it to the other highlighters and the results were exciting! It’s tempting to share those results but it’s definitely due for another benchmark, so we’ll work on that. Performance was the main motivator for creating the UnifiedHighlighter, as the standard Highlighter (the only one meeting Bloomberg Law’s accuracy requirements) wasn’t fast enough, even with term vectors along with several improvements we contributed back, and even after we forked it to highlight in multiple threads.
Migrated from LUCENE-7438 by Timothy M. Rodriguez, 8 votes, resolved Oct 07 2016 Attachments: LUCENE_7438_UH_benchmark.patch (versions: 2), LUCENE_7438_UH_small_changes.patch, LUCENE-7438.patch Linked issues:
5890