Currently heuristic faceting is controlled by a hitCount based limit facet.sparse.heuristic.fraction and a maxDocuments based facet.sparse.heuristic.sample.size.
The constant sample size means that the chance of errors with low hitCounts is high and the performance overhead with large hitCounts is unnecessarily high. Using a hitCounts based function for the sample size instead of a maxDocuments would remedy this.
If h is hitCount, d is maxDocuments and a & b are empirically determined constants, the formula a*(h/d)+c provides an output linear to hitCount over maxDocuments. Sanity checking by turning off heuristics when the sample size is > 50% and forcing a sample size >= 1‰ seems prudent.
Conservative: d = 100M, a = -1.0 and b = 1.0.
h = 1M ⟹ -1.0*(1M/100M)+1.0 = 99%` ⟹ full count
h = 10M ⟹ -1.0*(10M/100M)+1.0 = 10%` ⟹ full count
h = 20M ⟹ -1.0*(20M/100M)+1.0 = 20%` ⟹ full count
h = 50M ⟹ -1.0*(50M/100M)+1.0 = 50%` ⟹ full count
h = 75M ⟹ -1.0*(75M/100M)+1.0 = 75%`
h = 100M ⟹ -1.0*(100M/100M)+1.0 = 0%` ⟹ 1‰
Aggressive: d = 100M, a = -1.0 and b = 0.5.
h = 1M ⟹ 49%
h = 10M ⟹ 40%
h = 20M ⟹ 30%
h = 50M ⟹ 0% ⟹ 1‰
h = 75M ⟹ -25% ⟹ 1‰
h = 100M ⟹ -50% ⟹ 1‰
Empiric based example: d = 250M, a = -19 and b = 0.78.
Currently heuristic faceting is controlled by a hitCount based limit
facet.sparse.heuristic.fraction
and a maxDocuments basedfacet.sparse.heuristic.sample.size
.The constant sample size means that the chance of errors with low hitCounts is high and the performance overhead with large hitCounts is unnecessarily high. Using a hitCounts based function for the sample size instead of a maxDocuments would remedy this.
If
h
is hitCount,d
is maxDocuments anda
&b
are empirically determined constants, the formulaa*(h/d)+c
provides an output linear to hitCount over maxDocuments. Sanity checking by turning off heuristics when the sample size is > 50% and forcing a sample size >= 1‰ seems prudent.Conservative:
d = 100M
,a = -1.0
andb = 1.0
.h = 1M
⟹-1.0*(1M/100M)+1.0
= 99%` ⟹ full counth = 10M
⟹-1.0*(10M/100M)+1.0
= 10%` ⟹ full counth = 20M
⟹-1.0*(20M/100M)+1.0
= 20%` ⟹ full counth = 50M
⟹-1.0*(50M/100M)+1.0
= 50%` ⟹ full counth = 75M
⟹-1.0*(75M/100M)+1.0
= 75%`h = 100M
⟹-1.0*(100M/100M)+1.0
= 0%` ⟹ 1‰Aggressive:
d = 100M
,a = -1.0
andb = 0.5
.h = 1M
⟹ 49%h = 10M
⟹ 40%h = 20M
⟹ 30%h = 50M
⟹ 0% ⟹ 1‰h = 75M
⟹ -25% ⟹ 1‰h = 100M
⟹ -50% ⟹ 1‰Empiric based example:
d = 250M
,a = -19
andb = 0.78
.h = 1M
⟹ 70% ⟹ full counth = 10M
⟹ 2%h = 20M
⟹ -74%h = 50M
⟹ -302% ⟹ 1‰h = 75M
⟹ -492% ⟹ 1‰h = 100M
⟹ -682% ⟹ 1‰