namoray / nineteen

nineteen
7 stars 16 forks source link

Implement dynamic period score multiplier and stability assessment in scoring #74

Closed yvetteTsai22 closed 3 days ago

yvetteTsai22 commented 1 week ago

Overview

This PR introduces enhancements to the scoring system by refining how period scores and quality metrics are calculated. The main improvements involve introducing a more dynamic period score multiplier and incorporating stability measurements through the coefficient of variation (CV) of quality scores. These changes aim to make the scoring system more robust and responsive to the available data.

Related to Issue #70 .

Key Changes

  1. dynamic Period Score Multiplier:

The period score multiplier has been modified to be more flexible. Instead of a fixed value of 1 for sufficient data points, it now scales between 0.25 and 1 based on the number of available period scores, reflecting the data's abundance while still providing a baseline multiplier when data is scarce.

  1. stability Measurement with Coefficient of Variation:

A coefficient of variation (CV) is calculated for quality scores to assess stability. A lower CV indicates more stable scores, which can enhance confidence in the results. The CV is then transformed using exponential decay to ensure it remains within a usable range (0, 1]. The combined quality scores now incorporate the CV weights, which provide a boost to scores deemed stable

Benefits

  1. Flexibility: The dynamic multiplier adapts to the amount of available data, increasing proportionally with the number of recorded period scores until threshold.
  2. Stability Awareness: By incorporating the stability of scores into the final calculation, the system can prioritize more reliable data, leading to fairer and more consistent outcomes.
  3. Edge Case Coverage: The implementation addresses potential edge cases: The multiplier will never drop below 0.25 as before, ensuring that some score is always awarded. The use of exponential decay ensures that even minimal variations in stability will still contribute positively to the scores.

Please be noted that the changes have not been tested against various scenarios.

namoray commented 6 days ago

I'm not sure if this is intended to be a supporting PR - but this just changes how miners get their combined score - rather than how we select candidates for a query