fulmicoton commented 1 year ago

We have two ways to deal with range queries...

a DocSet that uses the scans the columnar and creates a buffer of the next valid docs. In its current version it is already quite sophisticated... The buffering window change size to try to adapt to different match ratio dynamically, and the buffering itself relies on SIMD. The multilinear codec probably hinders perf a lot however.
a filter at the collector level.

Experiment with the two solutions, and see if the filter solution outperforms the docset solution for most of the queries. If it is the case, we can then work on the QueryAST (once #3148 has landed) to bubble up range queries and extract range queries as filters.

trinity-1686a commented 1 year ago

3329 will add support for warming up a range of inverted index. This means there is a 3rd option, using classic tantivy RangeQuery over the inverted index. This is likely slower on large ranges, but likely faster on smaller ranges.

PSeitz commented 9 months ago

quickwit-oss / quickwit

Range query optim #3194

3329 will add support for warming up a range of inverted index. This means there is a 3rd option, using classic tantivy RangeQuery over the inverted index. This is likely slower on large ranges, but likely faster on smaller ranges.