search

lightvector / KataGo

GTP engine and self-play learning in Go
https://katagotraining.org/
Other
3.46k stars 561 forks source link

KataGo elo vs AlphaGo elo #193

Open bochen2027 opened 4 years ago

bochen2027 commented 4 years ago

Currently katago is at 1600 elo, what is that in the elo system that Deepmind used?

Is KataGo stronger than AlphaGo Fan, AlphaGo Lee, or how it compares to the Master and AGZ?

https://raw.githubusercontent.com/lightvector/KataGo/master/images/readme/jan2020vsjune2019.png

lightvector commented 4 years ago

No way to tell for sure. Deepmind has never publicly released their code or engine or network weights, nor have they released enough test data to make a proper statistical comparison.

Additionally, Elo systems are going to be a bit relative and incomparable, because the size of the Elo difference in Elo points between different bots actually changes when compute time or hardware changes, so usually you talk about Elo only in the context of having already pinned down the precise match settings and hardware that the bots will use.

Very likely KataGo and Leela Zero now are both much stronger than AlphaGo Lee was due to some analysis of the games and looking at the increasingly large gap between them and human pros. I would be personally also moderately confident that FineArt now is stronger than Master and AGZ were back when those two were showcased, although with no hard evidence of course. Beyond that, it's all speculation with no evidence. Given the high uncertainty, lack of data and the fact that Elo measurements are relative and often incompatible even if you had data unless you specify precise conditions, attempting to pin down numbers would be completely pointless.