search

tokee / lucene-solr

High cardinality faceting (SOLR-5894)
http://tokee.github.io/lucene-solr/
7 stars 1 forks source link

Add threaded counter updates #27

Closed tokee closed 9 years ago

tokee commented 9 years ago

Recent experiments with AtomicIntegerArray and AtomicLongArray suggests that they quite close to pure int[] and long[] performance. Together with the advance method on result sets this opens up for threaded counter updates within single fields.

This would mean a speed up of the counting phase roughly linear to free CPU power, for facets with a many (millions) of references from documents to values.

tokee commented 9 years ago

The ground work had been implemented in the experimental pack branch with the counters PackedOpportunistic and NPlaneMutable with the tank option. Unfortunately, preliminary experiments shows that some of the DocValues structures are not thread safe and thus require an instantiation per thread. This means that there's an overhead to running extra threads, at least if one starts multiple threads per segment.

tokee commented 9 years ago

Most of the underlying Solr structures uses ThreadLocal so with a shared executor, the overhead of extra instantiations is small. The bitmap branch has a working implementation.