tokee / lucene-solr

High cardinality faceting (SOLR-5894)
http://tokee.github.io/lucene-solr/
7 stars 1 forks source link

Speed up N-plane counter increments #33

Closed tokee closed 9 years ago

tokee commented 9 years ago

In order to avoid multiple identical entries in the tracker for sparse faceting, the incrementAndGet method is called in the underlying counter. For the tank implementation in NPlaneMutable uses the following code:

long original = get(index);
increment(index);
return original+1;

This is both (very low risk) error prone and highly inefficient. What is really needed is not incrementAndGet but isZeroAndIncrement as only the notification changes from 0 to something else are needed. Unfortunately this is not possible to do reliably without walking all planes with the current N-Plane layout. A slight change to the layout so that the first plane is always 1 if the full counter value is > 1 would solve this.

tokee commented 9 years ago

The proposed solution is described in detail in https://sbdevel.wordpress.com/2015/04/30/alternative-counter-tracking/

tokee commented 9 years ago

This has been implemented as nplanez counters in the bitmap branch, soon to be merged to trunk.