lenglund commented 9 years ago

Ido would like to discuss the ranking algorithm more in-depth to determine if it can/should be further fine-tuned for future use. Right now, many of the answer pairs receive the same score despite being compared 8-12 times.

[x] Refactor code base to package algorithm separately #283
[x] Investigate proposed algorithm or other candidates (ask Ido/James/Mark)
[x] Write prototype of algorithm chosen
[x] Replace existing algorithm with new one (Elo)

Possibly add ability to select algorithm? (issue #312)

lenglund commented 9 years ago

Michael will run some simulations to see what the best number of evaluations is for different group sizes.

lenglund commented 9 years ago

Conclusions:

Issues we’ve noticed exist regardless of number of evaluations, so 3 is fine as the standard
Difference in scores is not that great b/c each answer is involved in so few comparisons overall
In other words, there are not that many ratios to work with (number of wins vs. number of times compared)
Only way to fix this would be to put in weighting for the comparisons
As it exists, scoring works better for a smaller pool of answers
In this case, each answer is evaluated more times and there is a better chance that answers will have different ratios

We'd need to collaborate with researchers to implement weighting, should this issue go forward.

lenglund commented 8 years ago

Decision has been made to update the algorithm to include weighting, possibly using the Elo rating system (https://en.wikipedia.org/wiki/Elo_rating_system). Ido has the details for this.

andrew-gardener commented 8 years ago

Some observations of the algorithms after some simulations

Corrected Comparative Judgement Algorithm

Generally performed worse than Elo and True Skill
Recovered from incorrect selection in early rounds better than True Skill

Elo Algorithm

Takes fewer rounds to create unique ranks than Comparative Judgement
Better at creating even matches over time than Comparative Judgement
Recovers from incorrect selection in early rounds better than Comparative Judgement and True Skill

True Skill

Takes fewest rounds to create unique ranks
Best at creating even matches over time
Moderately better then Comparative Judgement and Elo at keeping mid ranked items ranked in the mid ranking range
Moderately better then Comparative Judgement and Elo at recovering from incorrect selections in later rounds
Very bad at handling early incorrect selections. Takes more rounds to recover
minor: Takes considerable longer to run algorithm from scratch (need to modify Score model to only calculate whats needed instead of everything)

andrew-gardener commented 8 years ago

We need to decide on some criteria for selecting with algorithm we want to go

andrew-gardener commented 8 years ago

301 Adds both Elo and True Skill algorithms with True Skill set as the default algorithm used. The DB will now also store the algorithm used for each score record in case we decide to change algorithms again or we give the instructor the option to select which algorithm they prefer.

andrew-gardener commented 8 years ago

Ido mentioned that:

Recovered from incorrect selection is important (in early rounds) since students are likely to make mistakes
Reaching unique ranks quickly is not important

Elo might be the better candidate given these criteria

ubc / compair

Change the ranking algorithm #241

301 Adds both Elo and True Skill algorithms with True Skill set as the default algorithm used. The DB will now also store the algorithm used for each score record in case we decide to change algorithms again or we give the instructor the option to select which algorithm they prefer.