The histogram portion of the beacon scoring attempts to judge the shape of the hourly connection count distribution numerically rather than leaving it for visual determination. Currently it detects single flat sections or uniform distribution by finding the coefficient of variation of the connection frequency table and detects dual flat sections by identifying bimodal cases in the histogram of the frequency table counts. The lack of jitter tolerance in this method has caused some valid beacons to be scored lower than they should have been.
This PR contains the following update to address this issue:
Instead of looking at single modes, we allow for some jitter by grouping them into buckets. Where before we were looking for 4 or fewer, we are now looking at how much of our data is captured in those modes instead of just counting modes.
The final score will be upgraded to take the maximum value of two subscores:
Histogram Score = max( cv-based score, bimodal fit score )
bimodal score = $max {1}(bucket) + max {2}(bucket) \over total\ bars - 1$
where $bucket$ is refers to the bars of the jitter-tolerant counts histogram and $total\ bars$ refers to the number of non-zero bars in the original frequency count table.
bucket size = $\lceil max _{1}(frequency\ bar) * 0.05 \rceil$
where $frequency\ bar$ refers to the connection count frequency table bars
Note: this score is only calculated if there are at least 10 $total\ bars$
Examples
Case 1: Uniform Distribution with Jitter and Gaps
In this case we have one mode and by bucketing our data we are hoping to find that mode. Since we are adding the two buckets together, the second one is redundant - it doesn't hurt but doesn't add anything. The cv score scored low in this case due to open gaps in the dataset and the original multimodal score scored low due to the jitter.
Case 2: Alternating Consistent Frequency
In this case our data is truly bimodal and we are going to identify it by combining totals from the two largest buckets in the frequency count histogram. The cv score will always score low for cases like this. Since there is no jitter, the new score is identical to the old version and shows that the update is sufficient in replacing the original version for bimodal detection.
Case 3: Uniform Distribution with Jitter, Gaps, and Unfortunate Bucket Split
All of our data fits into one mode, but it is a little bit noisy and because of the bucket size that we chose, it gets split into 2 buckets. In this case, we are taking the top 2 buckets which allows those two to be treated as one and scored appropriately. As with case 1, the cv score scored low due to the open gaps in the dataset and the original multimodal score scored low due to the jitter.
Case 4: Uniform Distribution with Outlier
In this case we have a uniform distribution and a big outlier. The coefficient of variance is affected by this outlier and scores lower than it should have. The bimodal detection handles this case well because it throws out one potential outlier, and while outlier detection could be added to the cv score, it may add unnecessary computation time since the new subscore handles this case.
Testing
C2 implant benchmarks:
No change for half, 9 had a score improvement (increase) of 1-2% and one had an increase of 16% (this one is represented in case 3 above).
False positive benchmarks:
No change for all except 1, which saw a score improvement of (decrease) of 4%
Problem
The histogram portion of the beacon scoring attempts to judge the shape of the hourly connection count distribution numerically rather than leaving it for visual determination. Currently it detects single flat sections or uniform distribution by finding the coefficient of variation of the connection frequency table and detects dual flat sections by identifying bimodal cases in the histogram of the frequency table counts. The lack of jitter tolerance in this method has caused some valid beacons to be scored lower than they should have been.
This PR contains the following update to address this issue:
Instead of looking at single modes, we allow for some jitter by grouping them into buckets. Where before we were looking for 4 or fewer, we are now looking at how much of our data is captured in those modes instead of just counting modes.
The final score will be upgraded to take the maximum value of two subscores:
Histogram Score = max( cv-based score, bimodal fit score )
cv score = ${\sigma \over \mu} 100$ = ${{standard\ deviation} \over mean} 100$
bimodal score = $max {1}(bucket) + max {2}(bucket) \over total\ bars - 1$ where $bucket$ is refers to the bars of the jitter-tolerant counts histogram and $total\ bars$ refers to the number of non-zero bars in the original frequency count table.
Examples
Case 1: Uniform Distribution with Jitter and Gaps
In this case we have one mode and by bucketing our data we are hoping to find that mode. Since we are adding the two buckets together, the second one is redundant - it doesn't hurt but doesn't add anything. The cv score scored low in this case due to open gaps in the dataset and the original multimodal score scored low due to the jitter.
Case 2: Alternating Consistent Frequency
In this case our data is truly bimodal and we are going to identify it by combining totals from the two largest buckets in the frequency count histogram. The cv score will always score low for cases like this. Since there is no jitter, the new score is identical to the old version and shows that the update is sufficient in replacing the original version for bimodal detection.
Case 3: Uniform Distribution with Jitter, Gaps, and Unfortunate Bucket Split
All of our data fits into one mode, but it is a little bit noisy and because of the bucket size that we chose, it gets split into 2 buckets. In this case, we are taking the top 2 buckets which allows those two to be treated as one and scored appropriately. As with case 1, the cv score scored low due to the open gaps in the dataset and the original multimodal score scored low due to the jitter.
Case 4: Uniform Distribution with Outlier
In this case we have a uniform distribution and a big outlier. The coefficient of variance is affected by this outlier and scores lower than it should have. The bimodal detection handles this case well because it throws out one potential outlier, and while outlier detection could be added to the cv score, it may add unnecessary computation time since the new subscore handles this case.
Testing