search

taralloc / tennis-prediction

Implementation of the paper "Machine Learning for the Prediction of Professional Tennis Matches"
42 stars 20 forks source link

Something wrong #2

Open mcekovic opened 6 years ago

mcekovic commented 6 years ago

There is clearly something wrong with this implementation, as simple ranking points predictor gives 67% prediction. Elo can get 68%, Surface Elo 68.5%. More sophisticated predictors can get up to 70%. But the inherent limit is somewhere around it.

Prediction % differs by season, surface, etc... Seasons in more dominant Eras are easier to predict. Grass is easier to predict then clay. Best of 5 prediction can reach 76%...

taralloc commented 6 years ago

Why do you think something is wrong with the implementation and not with the method itself?

Anyway, if you can spot the error, please share :)

mcekovic commented 6 years ago

Not sure what is wrong method or the implementation. But reading the original document, I did not get impression that method is clearly defined in details, and that a lot of stuff was open for interpretation (many parameters etc...). With the right features in input, theoretically neural network should get better results. Starting point should not be lower then simple predictor based on ATP rankings, which says better ranked player will win. Which is in ~67% cases in latest ATP history. However, the true value of the predictor is not prediction rate, but how accurate the probabilities outside of 50%-50% area are. Did not check the code, but I suppose the input features are not properly defined. For example, ATP ranking or ATP ranking points. Just having the rank or points in input is not good enough because when you apply the sigmoid or other function on raw ATP points or rank you get skewed probabilities outside of 50%-50% area. Input neurons should output probabilities, not plain values. These probabilities should be already calibrated so that calibration of the whole network is then much easier and tuning is faster. Also, depends on season you are measuring. Seasons in weak Eras are less predictable, seasons in strong Eras are more predictable. 2017 was weak, thus less predictable. Seasons 2000-2004 too. Differences in prediction rates between seasons can get up to 10%! You can check prediction rates throughout the ATP seasons for simple Ranking based predictor and for Elo rating predictor here: http://www.ultimatetennisstatistics.com/dominanceTimeline?predictability=true

zumpchke commented 3 years ago

What kind of useless thesis do not even mention the results or confusion matrix? Clearly author was fudging something..