Problems when playing against KataGo

[hope366]

Sabaki 0.43.3 KataGo 1.10.0 cuda kata1-40b s1033 4 stone handicap game (Komi = 0.5) Rule = Japanese rule

The screenshot below shows just before the end of the game.KataGo is making a "meaningless move" to put stones on his position. After this, even if I pass in black, KataGo puts a stone in his position. Why does KataGo behave like this when it's clear that there's nothing to move? This kind of phenomenon happens frequently, but is there any good solution?

[Friday9i]

KataGo can use 2 pass options (friendlyPassOk = true and conservativePass = true), did you deactivated them? But even if they are still activated, in the shown case, KataGo still wins by 20 points despite that move, so it's a bit surprising but it's not really a problem.

[hope366]

Thank you for your answer. I saw the "Passing and cleanup" section of default_gtp.cfg. There were two setting items, "# conservativePass = true" and "#preventCleanupPhase = true". Both are written as "true" by default in the description. I haven't changed any settings for this part, so both are "true". I can't find the "friendlyPassOk = true" you're saying, do I have to write it arbitrarily if I want to set this? excuse me. It was in the rules section. And it was "false".

[Friday9i]

That's probably the reason : -)

[lightvector]

No, none of these settings are likely to help.

• conservativePass only affects area scoring. It makes it so that if the opponent passes, KataGo won't try pass to win the game in an unfinished position according to strict scoring rules, since servers like OGS do not follow strict scoring rules and will still award territory for regions that are not completely surrounded, meaning that if KataGo passes in such a situation where strictly it should be winning, if OGS chooses to award the opponent's incomplete territory anyways and chooses not to award KataGo's territory, then it could be counted as a loss. This parameter prevents that.
• friendlyPassOk only affects area scoring. It makes it so that KataGo will be friendly and end the game a little quicker once the opponent starts to pass.
• preventCleanupPhase should always be left as true in games versus humans. It makes KataGo play a game the way humans play it - ending the play by agreement and following the GTP protocol to identify alive/dead stones. Instead of how KataGo trains - a cleanup phase where you actually capture all the dead stones (even in Japanese rules) as a way to prove what stones are alive/dead.

However:

• fillDameBeforePass - This is the setting that may be influencing this behavior. This make KataGo try to make sure to fill dame before passing, by making KataGo prefer to play certain kinds of moves before passing so long as doing so doesn't cost the win, and keeps the score similar.

I'm not sure you should set it to false though. If you do then KataGo might sometimes pass before filling a dame that forces the opponent to defend inside their territory. The reason is that for KataGo's particular formalization of the rules, the point that the opponent would fill would never be counted anyways - the opponent will always have to fill there eventually, even if both players would pass and then resume play. Servers like KGS do very well and amazingly often won't count such a point - if you know KGS, you'll see KGS will actually correctly omit points from false eyes and things like that even if those points aren't physically filled by the players, although it doesn't get all cases right. But servers like OGS definitely will count the point, and so KataGo needs to make sure to fill dame before scoring to force such moves.

But if there is a bit of noise with extreme scores (20 points is pretty large) then it might have some false positives and make it more likely to play a few unnecessary moves in addition to playing the dame before passing, I'm guessing this is what's happening.

@Friday9i - Thanks for helping to answer questions like this! I imagine you're pretty confused too about all the options. Please consider this post as your reference for the future, so that you can understand what these options do and better answer questions like this in the future if I'm not around. :)

[lightvector]

@hope366 Can you send me the SGF for testing so I can improve this in the future?

[hope366]

Thank you for your detailed explanation. The game record shown above has been discarded, so I don't have it at hand ... And I thought I could get a similar sample again soon, but this wasn't what I expected. I've tried it several times and it seems that KataGo rarely puts pointless stones on his position (even if there's a 20 point difference). However, it is often seen to put stones on the human position after filling all the Dame (movement that causes great damage if the human side does not take some action = movement that does not cause loss on the KataGo side).

We will let you know when we find a good sample. By the way, "fillDameBeforePass" doesn't seem to be found in the config file, where can I find this?

[lightvector]

Thanks!

By the way, "fillDameBeforePass" doesn't seem to be found in the config file, where can I find this?

There are parameters that are not present in the config file because I don't want to advertise to casual users that they exist, because they may be easy to set incorrectly and cause bad effects. This is one of them - I don't recommend that you try to change it. I am not going to provide any documentation for these parameters other than the raw source code.

Still, if you know for sure that a parameter exists, you don't need to "find" it, you can just put it into the config yourself and it will work. If you mistyped it, then KataGo's log file will give you the usual complaint that you provided an unrecognized parameter, but if the parameter truly exists, then it won't complain.

[michito744]

This is another case where KataGo cannot take care of the game correctly. It has an unusual restriction that white must win 10 pts. Even at the end of the game, KataGo was unable to calculate the number of the area correctly. This is because the network was not able to correctly determine whether or not to move additionally in the W's below area.

game record: (;GM[1]SZ[19]CA[Shift_JIS] PB[B_H5]BR[] PW[W]WR[] DT[2020-08-09] RE[W+10.0] KM[0.0]TM[]RU[Japanese] AP[Kiin Editor 1.11] AB[dd][pd][jj][dp][pp] ;W[fq];B[dn];W[nq];B[qn];W[pq];B[qq];W[qr];B[oq];W[pr];B[op] ;W[or];B[rq];W[np];B[lq];W[mr];B[no];W[mo];B[mn];W[lo];B[ln] ;W[nn];B[oo];W[on];B[qo];W[kn];B[ml];W[jo];B[eq];W[fp];B[km] ;W[jm];B[kl];W[jl];B[kk];W[qh];B[qf];W[qk];B[in];W[jn];B[hq] ;W[fr];B[go];W[fo];B[fn];W[gn];B[gm];W[en];B[fm];W[do];B[co] ;W[eo];B[cn];W[em];B[hn];W[cq];B[cp];W[dr];B[ck];W[dk];B[cj] ;W[cl];B[bl];W[dj];B[ci];W[ij];B[gj];W[ii];B[gh];W[dl];B[bm] ;W[eh];B[hk];W[ik];B[pk];W[pj];B[pl];W[oj];B[rk];W[ql];B[pm] ;W[fk];B[gk];W[rl];B[oc];W[fc];B[ec];W[fd];B[cf];W[hl];B[gl] ;W[qc];B[qd];W[jd];B[hc];W[hd];B[fb];W[gb];B[eb];W[df];B[de] ;W[cg];B[ef];W[dg];B[ee];W[bf];B[ce];W[mc];B[nb];W[gc];B[fj] ;W[di];B[ip];W[io];B[gp];W[jq];B[fg];W[bh];B[bi];W[ch];B[ai] ;W[be];B[bd];W[ad];B[ac];W[ae];B[bc];W[mb];B[ma];W[la];B[na] ;W[kb];B[rg];W[rh];B[bq];W[br];B[dq];W[cr];B[er];W[es];B[sh] ;W[si];B[sg];W[rj];B[kh];W[pf];B[pe];W[pg];B[hg];W[re];B[rd] ;W[rf];B[qe];W[qg];B[se];W[sf];B[sd];W[bp];B[bo];W[aq];B[iq] ;W[jp];B[ir];W[jr];B[ni];W[nj];B[nk];W[ok];B[ol];W[mk];B[nl] ;W[mi];B[mh];W[nh];B[li];W[oi];B[mg];W[le];B[kf];W[lf];B[lg] ;W[ke];B[rr];W[so];B[ro];W[jf];B[sn];W[sm];B[rn];W[fa];B[ea] ;W[ga];B[lk];W[mj];B[js];W[ks];B[is];W[oe];B[od];W[nc];B[pb] ;W[nd];B[kg];W[pa];B[qa];W[qb];B[ra];W[hf];B[gf];W[ge];B[if] ;W[ie];B[ig];W[rs];B[ao];W[lj];B[kj];W[ji];B[ki];W[fe];B[ff] ;W[fl];B[gs];W[gr];B[ap];W[bq];B[hm];W[il];B[ng];W[og];B[ne] ;W[nf];B[of];W[qp];B[rp];W[oe];B[sg];W[rg];B[of];W[lm];B[mm] ;W[oe];B[hs];W[fs];B[of];W[po];B[pn];W[oe];B[kr];W[ls];B[of] ;W[bk];B[ak];W[oe];B[ka];W[lb];B[of];W[al];B[am];W[oe];B[lr] ;W[ms];B[of];W[aj];B[bj];W[oe];B[qs];W[ps];B[of];W[oa];B[ob] ;W[oe];B[af];W[ag];B[of];W[fi];B[gi];W[oe];B[ek];W[el];B[of] ;W[mf];B[oe];W[ej];B[ho];W[me];B[rm];W[sl];B[gq];W[fh];B[sr] ;W[he];B[jg];W[je];B[ss];W[qs];B[hi];W[ih];B[ed];W[eg];B[hh] ;W[qm];B[ah];W[jh];B[cm];W[dm];B[hj];W[jk];B[im];W[hr];B[ep] )

[michito744]

supplement. The above game is not played by KataGo. This is an example of incorrect results analyzed by KataGo.

[lightvector]

@michito744 Thanks, interesting position. It doesn't seem related all to the problem described in this thread, but still interesting! The analysis is correct if you change the komi to -10, so it seems like this position has some subtle trickiness with horizon effect issues making with KataGo avoiding risk a bit too much when it doesn't matter for the game result. I will also consider this position when thinking about ways to mitigate horizon effect.

[michito744]

If I set Komi to -10, it converge on the correct conclusion in terms of win rate, but the phenomenon that score is incorrect does not change.

The above experiment was conducted under that condition.

[hope366]

I don't recommend that you try to change it.

@lightvector I understand well, thank you. It seems that the notification from GitHub did not come due to a bug in the mail software, so I was late to notice your reply ...

@michito744

It has an unusual restriction that white must win 10 pts.

Are these special conditions listed in the SGF? And will it be reflected when Lizzie or LIzzieyzy loads this SGF? When I actually load it, Komi is 0.0, so it does not seem to be reflected. Do I need to manually set Komi to -10? I set Komi to -10. KataGo is not trying to calculate because there is nothing to move in the final phase, but in the situation just before the final phase diagram, it is almost equal as shown in the screenshot above, and KataGo can calculate accurately. It looks like it's done ... Black: Territory = 41 Number of captured stones = 23 White: Territory = 51 Number of captured stones = 23 The result is even because the Komi is -10.

If you just load the SGF you showed, the Komi is 0.0. When I let KataGo analyze the final phase diagram, it looks like the screenshot below. White is 10 points ahead on the board, but it looks like he thinks he should make unnecessary moves to win more safely and reliably. Does this mean that you say "KataGo is not able to calculate accurately"?

[lightvector]

@michito744 - As you can see from hope366's screenshot, the winrate AND the score are both correct when komi is -10. It is only when komi = 0 and the game result is fairly extreme that the score is incorrect by 1 point.

This is definitely something that KataGo can improve on, thank you for providing the example. I am pretty sure it it is a separate problem from the original problem in this thread that hope366 reported. I think hope366's original problem will be a bit easier to solve, if they can find an example position for me to test, since it is likely just a straightforward bug in one of the heuristics in the code.

For your 1 point difference with 0 komi, I think it will be hard to solve, because it is not a bug in the code, it is sort of like KataGo's neural net saying "I am so far ahead, I don't want to bother reading just in case I have a blind spot, I will protect just in case and win, it is only 1 point", and even though it is a simple reading it is hard to fight against the neural net's instincts. I will think about how to improve it.

[michito744]

@hope366

I don't think LizzieYzy reflects the right ruleset. In the sgf file, Komi = -10 is set. If you run KataGo on Lizzie with Japanese rules, you will end up with a score like this.

[michito744]

@hope366 I'm sorry, I uploaded an incorrect file.