search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
7.31k stars 298 forks source link

Refactor file source #5100

Closed rdettai closed 1 week ago

rdettai commented 1 month ago

Description

To prepare the S3 backed SQS source, this PR refactors the file source so that generates RawDocBatch can be reused.

How was this PR tested?

Added extra unit tests to the new DocFileReader.

github-actions[bot] commented 1 month ago

On SSD:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 2337, ref commit: 4ade7b5ec59685fc508c1834f2e23b5ca7b5afbe
Link

On GCS:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 2339, ref commit: 4ade7b5ec59685fc508c1834f2e23b5ca7b5afbe
Link

rdettai commented 1 week ago

Closing this in favor of https://github.com/quickwit-oss/quickwit/pull/5148 that brings significant changes on top of the file source,