This includes a new scoring system discussed at our 2016/07/06 meeting. Previously, our scoring has worked this way:
assign sibling-relative weights to attackers, then sibling-relative weights to categories, then to types of criteria (quality/feedback/usability), then to individual criteria. The final weight of a criterion is achieved by finding the fraction represented at each of the four categories and cross-multiplying.
The new system will instead be based on the threat model categories instead of the criteria categories. The four categories are: attacker, attack, countermeasure, criteria.
Attackers and attacks will be sibling-relative weighted as before. A countermeasure’s score will be the percentage to which it successfully defends/mitigates an attack’s access to private information. Same for criteria, except that criteria can have a variety of arithmetic relationships to each other, and so may optionally be broken into logical groups. For example, Criteria A and B may each satisfy Countermeasure C 50% each, but not in an additive fashion, and so a wallet implementing both will still be capped at satisfying the countermeasure 50%.
The weighted score for a criteria will thus be determined by taking the weighted average of attackers and attacks as before, and multiplying them by the percentages in lower categories (countermeasures, criteria/criteria-groups). Percentages can never exceed 100%, and will be normalized to a max of 100% after multiplication.
This includes a new scoring system discussed at our 2016/07/06 meeting. Previously, our scoring has worked this way: assign sibling-relative weights to attackers, then sibling-relative weights to categories, then to types of criteria (quality/feedback/usability), then to individual criteria. The final weight of a criterion is achieved by finding the fraction represented at each of the four categories and cross-multiplying.
The new system will instead be based on the threat model categories instead of the criteria categories. The four categories are: attacker, attack, countermeasure, criteria.
Attackers and attacks will be sibling-relative weighted as before. A countermeasure’s score will be the percentage to which it successfully defends/mitigates an attack’s access to private information. Same for criteria, except that criteria can have a variety of arithmetic relationships to each other, and so may optionally be broken into logical groups. For example, Criteria A and B may each satisfy Countermeasure C 50% each, but not in an additive fashion, and so a wallet implementing both will still be capped at satisfying the countermeasure 50%.
The weighted score for a criteria will thus be determined by taking the weighted average of attackers and attacks as before, and multiplying them by the percentages in lower categories (countermeasures, criteria/criteria-groups). Percentages can never exceed 100%, and will be normalized to a max of 100% after multiplication.