search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
6.99k stars 291 forks source link

remove hits beyond max requested hit #5180

Closed trinity-1686a closed 1 week ago

trinity-1686a commented 1 week ago

Description

due to the 1st case of what's described in https://github.com/quickwit-oss/quickwit/issues/3650 , it can happen that we return up to start_offset + 2 * max_hits documents instead of max_hits. This happens when a split first fails, and then succeed, but its result is just concatenated instead of doing a proper top-k. The fix consist in dropping the tail hits at the same time as the hits that should be omitted due to start_offset are removed.

How was this PR tested?

tested with a modified s3 that errors on a fraction of requests. Without the patch, i often get too many results (and sometime a double error so no response), with the patch, i either get the right number of docs, or said error.

github-actions[bot] commented 1 week ago

On SSD:

Average search latency is 0.995x that of the reference (lower is better).
Ref run id: 2374, ref commit: f4e57102bf98ea1acff2feb2bc9fcf345f64567d
Link

On GCS:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 2375, ref commit: f4e57102bf98ea1acff2feb2bc9fcf345f64567d
Link