search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.09k stars 330 forks source link

Improve merge performance. #4131

Open fulmicoton opened 11 months ago

fulmicoton commented 11 months ago

Materializing the document mapping and (more importantly) doing lookups is not necessary.

        let mut merged_doc_id_map: Vec<Vec<Option<DocId>>> = self
            .readers
            .iter()
            .map(|reader| {
                let mut segment_local_map = vec![];
                segment_local_map.resize(reader.max_doc() as usize, None);
                segment_local_map
            })
            .collect();
     ...
fulmicoton commented 11 months ago

It might actually be negligible.