The idea here is to scientifically measure (and graph!) how accurate the match score predictions in the app are, as a function of how many matches have been played so far in the regional.
So, after the season is over we will have access to all the match scores from all of the regionals. We can write a simple program to only hand the prediction engine the scores for the first 10 matches, then see what it predicts for the score of the 11th match and compare that to the actual score. Then give it the 11th match and see what it predicts for the 12th, and so on. Graph it, with error bars!
With data from all the regionals in the world, we should be able to come up with pretty darn good error bars that measure how accurate the prediction is, and at what point in the regional we can start trusting it.
For bonus points we try to come up with (or find on Chief Delphi) a bunch of different prediction algorithms (the one that's currently in there is: add up the OPRs of the teams in the match) and compare the different algorithms against each other by how well they perform on real regional data!
This would make a pretty sweet science fair project, if anybody's doing a science fair. We could definitely publish a paper on Chief Delphi, and possibly somewhere more official if we get good results.
wall-daniel
commented
6 years ago
The idea here is to scientifically measure (and graph!) how accurate the match score predictions in the app are, as a function of how many matches have been played so far in the regional.
So, after the season is over we will have access to all the match scores from all of the regionals. We can write a simple program to only hand the prediction engine the scores for the first 10 matches, then see what it predicts for the score of the 11th match and compare that to the actual score. Then give it the 11th match and see what it predicts for the 12th, and so on. Graph it, with error bars!
With data from all the regionals in the world, we should be able to come up with pretty darn good error bars that measure how accurate the prediction is, and at what point in the regional we can start trusting it.
For bonus points we try to come up with (or find on Chief Delphi) a bunch of different prediction algorithms (the one that's currently in there is: add up the OPRs of the teams in the match) and compare the different algorithms against each other by how well they perform on real regional data!
This would make a pretty sweet science fair project, if anybody's doing a science fair. We could definitely publish a paper on Chief Delphi, and possibly somewhere more official if we get good results.