search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.69k stars 1.04k forks source link

Make BlockPackedWriter constructor take an acceptable overhead ratio [LUCENE-5265] #6329

Closed asfimport closed 7 years ago

asfimport commented 11 years ago

Follow-up of http://search-lucene.com/m/SjmSW1CZYuZ1

MemoryDocValuesFormat takes an acceptable overhead ratio but it is only used when doing table compression. It should be used for all compression methods, especially DELTA_COMPRESSED whose encoding is based on BlockPackedWriter.

Migrated from LUCENE-5265 by Adrien Grand (@jpountz), resolved Aug 22 2017 Attachments: LUCENE-5265.patch

asfimport commented 11 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

+1

I was also wondering if we could somehow, if the acceptable overhead ratio is "high enough", not use any blocking and just make a single packed ints array. This is even more RAM wasting, since you lose the per-block compression, but w/ a high acceptable overhead ratio, that's the idea ... or, we could make a separate DVF for this case.

asfimport commented 11 years ago

Robert Muir (@rmuir) (migrated from JIRA)

+1

asfimport commented 11 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

not use any blocking and just make a single packed ints array [...] or, we could make a separate DVF for this case

+1 I think it would be easy to do it in the current MemoryDVF.

asfimport commented 11 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Here is a patch.