How can I check uncertainty measure?

[phdsmhong]

Hello, thank you for creating Katago. I have benefited tremendously.

I heard that Katago evaluates an estimate of how uncertain or inaccurate the winrates and scores are. I was just wondering whether there is a way to get the uncertainty measure.

I could find the uncertainty measure for smaller boards, but I am looking for the measure for 19*19 board. Can you please advise? https://katagobooks.org/book7tt/root/root.html

[lightvector]

Thanks!

So, for uncertainty, there are a few different things, depending on what you're looking for: https://github.com/lightvector/KataGo/blob/master/docs/Analysis_Engine.md

As documented in the above link, one thing you can look at is the standard deviation of the ownership, such as via includeOwnershipStdev or includeMovesOwnershipStdev, if you're interested in the uncertainty about the particular territories.

Another thing you can look at is the scoreStdev, which reports the uncertainty of the score of the final score. Although, note that this number is biased a little high due to how MCTS works. This number is often the largest early in the opening, because early in the opening is the farthest away from the end of the game and so the uncertainty in what the very final score will eventually be is the largest.

If you're interested in the raw neural net's uncertainty about moves, you can look yourself at whether the raw policy probabilities are focused only on specific moves, or are spread out between many moves. Similarly you can look at how spread out the visits are.

If you're interested in the short-term uncertainty about the value of the position currently there are some numbers computed by the net but that are not output right now. I pushed a change just now that will go out in next release that will report some of these values. https://github.com/lightvector/KataGo/commit/05ec5a45e70caf4a9f1552858980651603fb33d9. The short term uncertainty is the guess of the raw neural net about how much its prediction might differ from its position after a few turns. If you can build the master branch and compile KataGo yourself you can try it out now.

[phdsmhong]

@lightvector thank you so much for your kind explanation. I think the short-term uncertainty measure is what I am looking for. I am thinking of classifying Go players' styles (e.g. whether a player is risk-averse or risk-loving). I will be waiting for the next release! :)

Unfortunately, I am not very good at coding. I use Pytorch, but I am having a hard time in loading your pretrained weights that I found in katagotraining.org. Do you have any advice?

[lightvector]

The pretrained weights are not in pytorch, they're in tensorflow (and specifically, the ancient and outdated TF 1.15). KataGo will be updating to use pytorch by the end of this year though.

What are you trying to do, exactly? You could try manually compling KataGo (for example on linux, it shouldn't be that hard, windows may take a bit more work) - https://github.com/lightvector/KataGo/blob/master/Compiling.md - and take advantage of the change I just pushed out. Once you have that, you can run the analysis engine using python's subprocess and send json queries to KataGo and read off the new fields from the reply: https://github.com/lightvector/KataGo/blob/master/docs/Analysis_Engine.md