search

tokee / lucene-solr

High cardinality faceting (SOLR-5894)
http://tokee.github.io/lucene-solr/
7 stars 1 forks source link

Tune packed/not-packed based on field layout #18

Closed tokee closed 9 years ago

tokee commented 9 years ago

If the number of unique values in the field is relatively low and there are many references to those, this will result in a lot of updates of a small amount of memory. This is best handled by the CPU-easy and cache-unfriendly SparseCounterInt, as opposed to the CPU-intensive and cache-friendly SparseCounterPacked.

Performance testing with different uniqueValues and references should be done in order to determine which sparse counter to use in different setups.

tokee commented 9 years ago

In the current experimental branch pack, a histogram of maxima is available for most implementations. This can be used for better predictions.

tokee commented 9 years ago

This is now controlled via facet.sparse.counter=auto and facet.sparse.packed.bitlimit=X.