Vandertic
commented
4 years ago You are right, that our promotion procedure is an approximate one. If we had wanted to choose the best candidate, we should have really being doing a lot more match games, as 60 or so are not nearly enough. Maybe we should have also made the candidates play one another. The idea behind these matches, though, is to just have a quick peek to the performances of candidates, in order to avoid choosing a really bad one. Good or best is the same for us. AlphaZero did not perform any kind of gating and still got superhuman. In fact, I am really thinking about decreasing a little the amount on matches, since they are using quite a lot of resources and now that the level is high enough, there is really not much difference between one candidate and the other. Or at least, the differences do not reflect enough in match games performances.
as we know, leelazero use 400 games for a match and 0.55 gate to pass, sometime it will pass earlier when the winrate is high at less games, such as 360 games and 0.58. they should be similar in probability of pass. and indeed, the winrate may drop down to 0.51 for 800 games. as to sai, when we move to 9x192 games, the amount of a match often exceed 60 due to some slow computer did not finish in a long period, so the server dispatched more games. is it fair to promote only according to the final score?
eg.
according to the final result , 0c5cc649 should pass. but in the match of d615b6b4, the winrate was 55.47 at 64 games, same as that of 0c5cc649 match. back to 50 games, d615b6b4 got 62, and 0c5cc649 got 53..