lightvector / KataGo

GTP engine and self-play learning in Go
https://katagotraining.org/
Other
3.49k stars 564 forks source link

Cannot reproduce b28 rating game resign conditions #937

Closed Ishinoshita closed 4 months ago

Ishinoshita commented 4 months ago

Hi lightvector,

Maybe more a curiosity than a real issue.

Like many people these days, I have had a look at b28 net rating games on katagotraining.org to have a feel about how differently it plays vs b18 nets. The second game I scrolled through stroke me:

Rating Game [ID 1285683](https://katagotraining.org/sgfplayer/rating-games/1285683/)

The sgf data show both net agreeing on a solid 4 pts score lead for White and a win rate over 95% for the latter (b28): image

However, with my modest kyu level (KGS ~5k), it was not immediately obvious why the game was resigned. I thought I might be badly missing something related to groups status, so I downloaded the sgf and analyzed it with Katrain, with my current b18 net. To my surprise, my b18 evaluated the end position as not so decided (circa 70%for White, b28). Could the b18 and b28 that played the actual game have such a different opinion?

I downloaded both nets used for that match and, to my surprise again, I was not able to see the obvious resign condition, with any of them (tried different visits level, from 500 to 1500, close to the match conditions; tried to clear cache and redo analysis several time: win rate fluctuated only a few % around 67%): image image

I have a vague remembering of match resigning conditions like this: image but I don't know what parameters apply to rating game (was unable to locate the config in the code on github).

NB: I did have a look at a dozen b28 rating games and was not able to find another example.

Another curiosity: why should game length parity (odd/even number of moves when game ends) correlate to network color? For example, here are 100 games (one page) of b28 rating games: image

Probably a good explanation but that I cannot see ! :-)

lightvector commented 4 months ago

Do you have the correct rules set? This game has tax=ALL. White has 1-2 groups, while black has 4 groups, so White will be relatively more ahead in group tax rules than in normal rules.

For parity, this is just resignation, right? A player checks whether they want to resign only on their own turn. (A bit of a quirk is that the MCTS search that sees the final consecutive winrate to resign also generated a move, so the SGF may still contain the move of the resigning player, instead of the resignation replacing the move).

Ishinoshita commented 4 months ago

I fell for the rule set trick sorry! I thought that all rating games are played with the same rule set, thought it was Chinese. So I didn't pay attention...

Re game length parity, if one assume a much stronger net, it make sense to assume that it has a better understanding of the game status and will be each time the first to initiate the resignation check cycle. I was simply not expecting 18b to be that systematically behind in this evaluation.

In fact, did observed the same super strong correlation for a recent 18b net (rating games still include lot of games vs weaker 40b/60b where the same thing happens).

Thanks for your clarification 👍