asfimport
commented
9 years ago David Smiley (@dsmiley) (migrated from JIRA)
This first patch is the first step / phase, which differentiates "covered" leaf cells (cells that were within the indexed shape) from other cells which are approximated leaf cells. The next phase will be augmenting the Intersects filter and possibly others to collect exact hits in conjunction with the fuzzy hits.
During the benchmarking I learned some interesting things:
- Quad tree is 50% of the size of Geohash! This observation is for non-point data, since that's what's relevant to all this hit-confirmation business / leaf cells. For point data, it'd be the other way around.
- Leaf pruning shaves 45%! So much for my plans to phase that out – it's key.
- Differentiating leaf types (Covered vs Approximated) add 4%.
- A more restrained leaf pruning that doesn't prune covered leaves larger than those at the target/detail level yields 36% shaving (not as good as 45% – expected). That is... we're adding these covered leaf bytes to subsequently make exact results checking better so we don't want to be too liberal in removing them. There's a trade-off here.
The attached patch includes some refactoring to share common logic between Contains & AVPTF (the base of Within, Intersects, and heatmap). I need to add a configurable flag to indicate if leaves should be differentiated in the first place, since you might not want that, and another flag to adjust how much pruning of the covered leaves happens. Both flags should be safe to change without any re-indexing; it could be changed whenever. Obviously if you don't have the covered leaf differentiation then you won't get the full benefit later when we have exact match collection, just partial.
If a cell is within the query shape (doesn't straddle the edge), then you can be sure that all documents it matches are a confirmed hit. But if some documents are only on the edge cells, then those documents could be validated against SerializedDVStrategy for precise spatial search. This should be much faster than using RPT and SerializedDVStrategy independently on the same search, particularly when a lot of documents match.
Perhaps this'll be a new RPT subclass, or maybe an optional configuration of RPT. This issue is just for the Intersects predicate, which will apply to Disjoint. Until resolved in other issues, the other predicates can be handled in a naive/slow way by creating a filter that combines RPT's filter and SerializedDVStrategy's filter using BitsFilteredDocIdSet.
One thing I'm not sure of is how to expose to Lucene-spatial users the underlying functionality such that they can put other query/filters in-between RPT and the SerializedDVStrategy. Maybe that'll be done by simply ensuring the predicate filters have this capability and are public.
It would be ideal to implement this capability after the PrefixTree term encoding is modified to differentiate edge leaf-cells from non-edge leaf cells. This distinction will allow the code here to make more confirmed matches.
Migrated from LUCENE-5579 by David Smiley (@dsmiley), 1 vote, resolved Apr 10 2015 Attachments: LUCENE-5579_CompositeSpatialStrategy.patch (versions: 2), LUCENE-5579_SPT_leaf_covered.patch, spatial.alg Linked issues:
7423