search

harbecke / HexHex

AlphaGo Zero adaptation for Hex
GNU General Public License v3.0
20 stars 4 forks source link

creating elo ratings takes too long for many models #31

Open harbecke opened 4 years ago

harbecke commented 4 years ago

hex/elo/elo/create_ratings takes approximately n²/10000 seconds to calculate for n models on the server. this is way too much for long trainings and should rather be an approximation than quadratic time.

harbecke commented 4 years ago

is quickfixed with fd76242 - no more elo calculations

solving the problem seems to be difficult, i took a quick look at https://cran.r-project.org/web/packages/BradleyTerryScalable/vignettes/BradleyTerryScalable.html which is based on https://arxiv.org/abs/1011.1761 but that doesn't solve that each run takes a long time

maybe we should run the elo calculation once at the end of the training to determine the best model? @cleeff @simonant any ideas?

cleeff commented 4 years ago

Hmm that's unfortunate. I remember that I encountered this already and in elobayes[1] also an incomplete game graph is allowed. You can have each agent play a fixed number of games against a random subset or the best agents so far. The numbers aren't very accurate in that case but probably good enough.

Having agents being evaluated 'at the end' / after training sounds like this could easily be missed if you want to intermediately abort training.

[1] elobayes is the tool we previously used and which is also used in the AlphaGo papers.

harbecke commented 4 years ago

sorry for being unclear. we already have an incomplete graph (we let each new model play against the best m=3 models so far), however the calculation of the elo list itself takes a very long time (we have up to 5000 models during a complete training run).

i think this is a low priority issue, as the reference models are more informative for the progression anyway.

cleeff commented 4 years ago

Ahh that makes more sense. Yes, I totally forgot about that. I agree let's keep the elo computation out for now.