Strong 10b KataGo net

[Friday9i]

Update 2020-01-11: new stronger 10b network published in a comment below, around 20 Elo stronger than the previous one, around 200+ Elo stronger than the official g104 10b and close in strength to g104 15b (much better on small boards, a bit weaker on 19x19)

Update 2019-12-20: I just added a comment below, with a significantly stronger 10b network This net seems to be ~50 to ~100 Elo better than the previous 10b net on board sizes from 9x9 to 19x19 (but possibly a bit weaker for sizes 7x7 and 8x8). I did not test it against LZ, but on time parity with Cuda, this net must be strong (as 10b128c nets are very fast : -).

Hello I'm doing an independant selfplay + training run of KataGo nets, simultaneously doing selfplay with a 10b128f and a 30b256f net, and training them in parallel ; -). Recently, I added a 15b192f (still weaker than the 10b net but coming close). I began from random nets and was doing selfplay with extremely low visits: only 5 cheap and 30 normal visits! And it worked well until the 30b net reached a level approximately in the middle between the official g104 10b and 15b nets (ie probably around ~LZ120's strength). Then it stagnated for some time, so I upped the visits progressively to 30 cheap / 120 normal visits. And as imrpovements are slowing down, I'm upping visits to more classic 100/600 or more visits... 30b was improving again but 10b stagnated until I decreased the LR to 0.3 (it gained ~100 Elo very quickly). So I'm currently reducing the LR of 10b progressively while I keep the 30b LR at 1 as it still improves well. Today, after 1.5 months in the run (on 2 GPUs: RTX2070+RTX2080), here are the results:

Trained 10b net: It is ~200 Elo stronger than the official g104 10b net on all board sizes from 6x6 to 19x19. Here are the detailed results running gatekeeper with random odd komi (within ~1 point from fair komi) and up to 200 games for each board sizes, with 100 visits: 6x6 | 7x7 | 8x8 | 9x9 | 11x11 | 13x13 | 19x19 69% | 81% | 89% | 72% | 71% | 75% | 80% With more visits, results would probably be even more pronounced in favor of this net (but I didn't test it). This net is also better than g104 15b (and much faster of course!) on board sizes from 6x6 to ~10x10, and probably still competitive up to 19x19 on time parity. Here are the results with 100 visits for both nets (on same board sizes): 67% | 78% | 75% | 62% | 41% | 39% | 20% Here is the 10b net (I'll publish new versions in the coming weeks if they get significantly stronger): model.txt.gz

Trained 15b net: it is very recent and still a bit weaker than 10b (hence little interest for the moment) Trained 30b net: it is clearly stronger than the 10b, especially on boards bigger than 13x13, but still "only" around g104b 15b strength (ie close to LZ145/LZ150) while much slower. BTW, it's just amazing to generate from scratch a net around LZ145's strength on a single computer in 1.5 months!!! I'm now switching to a mix training and selfplay to see if it accelerates the training (using 10b for deep selfplay games mainly on small board sizes while I generate 100/600 visits 30b games mainly for large boards sizes): we'll see if it is efficient!

[cryptsport]

@Friday9i thanks! very interesting! 30b256f is not a typo? 30b or 20b?

[Friday9i]

Not a typo, it's a 30b256f network indeed ; -) 20b256f already exists, so I thought it would be interesting to test a slightly larger net and see if it can go higher. And it's still manageable in terms of size and speed: it takes around 2 hours to train it, and it plays reasonably fast in selfplay, so an attractive size I think.

[cryptsport]

@Friday9i yes it is interesting, thanks!

[kaorahi]

Great for CPU-only users! Can I use it as a build-in network of my GUI?

If possible, would you provide saved_model/{model.config.json,variables} like the official g104? They are required for libtensorflow backend (https://github.com/lightvector/KataGo/issues/14#issuecomment-549142392).

[Friday9i]

Unfortunately I had a crash and went back to a backup 24 hours old, and I lost this folder... Good news is I have a more recent 10b which seems close in strength to the previous one (maybe slightly weaker, but within noise, so about comparable). Here is the full zipped folder (too big, so here is a link): https://filetransfer.io/data-package/YiGnULsu (and maybe you can try to mix the saved_model from this net and the net provided 2 days ago, it may work, or not... Hopefully in the coming days I'll get clearly better 10b nets, so I'll upload the nets and associated files of course). If you have the opportunity to test this net, I'll be happy to get some news ; -)

[kaorahi]

Thanks! I got converted *.pb from Dec8 version successfully for libtensorflow backend.

I cannot combine Dec8 version with model.txt.gz in Dec5 version. The converter seems to depend only on saved_model/{variables*,model.config.json}. Anyway, I will try Dec8 version as my default network for the game analysis for a while.

Would you permit to use / distribute your networks as built-in ones inside a packed *.exe? I will refer to the origin of them in the release notes like this.

[Friday9i]

Cool ; -) And yes of course, you can use and redistribute the nets I trained! (if lightvector is ok also, as it is done with KataGo of course)

[lightvector]

Seems okay to me. It's all open-source anyways. :)

[kaorahi]

Then I will use Dec5 version in the next release. I enjoy watching its wins against ELFv1 in time parity games on my machine. Thanks.

[l1t1]

useful work

[l1t1]

i am running a test D:\tool\go_gui\gogui-twogtp -white "D:\KataGo-1.1\katago12_cuda gtp -model d:\model1205.txt -config D:\gtp_example.cfg" -black "D:\KataGo-1.1\katago12_cuda gtp -model d:\model1209.txt -config D:\gtp_example.cfg" -games 20 -sgffile katago12cuda_120509 -auto -komi 7.5

#GAME   RES_B   RES_W   RES_R   ALT DUP LEN TIME_B  TIME_W  CPU_B   CPU_W   ERR ERR_MSG
0   B+R B+R B+R 0   -   337 2286.7  2284.4  0   0   0   
1   W+R W+R W+R 0   -   192 1152.9  1082.4  0   0   0   
2   B+R B+R B+R 0   -   197 1191.8  1163.1  0   0   0   
3   B+R B+R B+R 0   -   145 1636.1  1777.1  0   0   0   
4   B+R B+R B+R 0   -   191 2613.5  2614.5  0   0   0   

[Friday9i]

@l1t1 Gatekeeper is way faster than Gogui for comparing KataGo nets : -). You can use the following text in a bat file (on windows): "C:\katago gatekeeper -rejected-models-dir C:/RejectedNets -accepted-models-dir C:/AcceptedNets -sgf-output-dir C:/SGFs -test-models-dir C:/TestedNets -config-file gatekeeper.cfg >> Result.txt" And you put the net you want to test in "TestedNets" and the reference net you want to compare to in "AcceptedNets" (iirc correctly, it should be older than the candidate net as gatekeeper is ... a gatekeeper. So take that into account). You can use ~100 to ~500 threads and simultaneous games with an RTX card.

[l1t1]

@Friday9i thanks

[kaorahi]

very aggressive H4 game by Dec5 network (W, 30sec) against leela 0.11.0 (B, 10sec) with modified katago (https://github.com/kaorahi/KataGo/tree/keep_uneasy)

L0110_H4.sgf.txt

[Friday9i]

2019-12-20: here is a link to a new 10b apparently significantly stronger: https://filetransfer.io/data-package/kxPmeRyR And the model itself (without additional folders) is here: model.txt.gz

@kaorahi You can of course use this new one too : -)

[kaorahi]

Thanks again :) I use your network by default for game analyses now and I will pack it into the next release of my GUI. It is good at large capturing races for a 10b network in my impression.

[Friday9i]

Nice! If you run some benchmarks / matches against other nets (LZ, Elf, other KataGo nets ...), would you mind to share the results here? Thanks a lot

[kaorahi]

My PC is too slow for benchmarks... I have only a portable laptop without GPU. (So I appreciate your small & strong networks.)

[cryptsport]

gt 610 is also not a strong card, but maybe it's interesting: 50 games KataGo 10x128-s1108 - KataGo 10x128-s4073 (gt 610, 5 sec/move), new network won 42:8 (21 b and 21 w) 50 games KataGo 15x192-s2973 - KataGo 10x128-s4073 (gt 610, 5 sec/move), new network won 32:18 (15 b and 17 w) 50 games KataGo 20x256-s4479 - KataGo 10x128-s4073 (gt 610, 5 sec/move), new network lost 8:42 (1 b and 7 w)

[Friday9i]

Thanks a lot @cryptpark Are those games only on 19x19 with 7.5 komi, or spread on different board sizes? How many visits approximately with 5s/move? Anyway, it seems to confirm this first 10b published (s4073) is well above the official g104 10b net and also slightly above the official g104 15b net ; -). (and it is much better on boards below ~13x13) And the last net published yesterday (s4588) should be even better, hopefully somewhat closer to g104 20b on 19x19. My aim now is to make a 15b net better than g104 20b net on time parity : -). We'll see if I manage to reach that! And I'll still train the 10b net to see if I can improve it a bit more.

[cryptsport]

@Friday9i Yes, on 19x19 with 7.5 komi. first 20 moves Root visits (gtp.log) - min 147 to max 789 (10x128-s4073). in an hour I will start the match s4588 - 20x256-s4479. Thanks again for your interesting work! and good luck !!!

[cryptsport]

It is fantastic! new network on the same level with 20x256! now 19:23

[Friday9i]

Almost too good to be true! Let's wait for some more games, but that's a good start ;-)

[cryptsport]

50 games KataGo 10x128-s4588 - KataGo 20x256-s4479 (gt 610, 5 sec/move), new network lost 22:28 (8 b and 14 w). s4588 white won 14-11!

[l1t1]

suggest release it as a new offical weight, so that we can have a new benchmark just like http://www.yss-aya.com/cgos/19x19/cross/kata12_104b20_v800.html

[cryptsport]

@Friday9i I want to make a match against SAI 9x9. What parameters to use for maximum KataGo power?

[Friday9i]

With a good GPU, you should use 10 to 20 threads, but for a limited GPU, 3 or 5 threads may be better. And equivalent time I would say

[cryptsport]

and -r, -b and other options?

[Friday9i]

The -r option is managed by the config.gtp file, and I don't know -b : -)

[cryptsport]

it's from LZ, I thought the same thing in KataGo. Thank you!

[cryptsport]

100 games 9x9, KataGo 10x128-s1108 - SAI (gt 610, 5 sec/move) 30:70 (KataGo b 9:41 and KataGo w 21:29) 100 games 9x9, KataGo 10x128-s4588 - SAI (gt 610, 5 sec/move) 65:35 (KataGo b 23:27 and KataGo w 42:8)

[Friday9i]

Eh eh, cool ; -). Which version of SAI?

[cryptsport]

the latest release of SAI, 0.17.4-gpu https://github.com/sai-dev/sai/releases/

Once again I want to say - KataGo - a fabulous program! And of course, new networks add joy!

[22nsuk]

@Friday9i How's the progress going on for 30 blocks? I want you to release this as well. And I'd appreciate it if you could summarize and explain how to train katago. Is that possible? Thank you for sharing a good model.

[cryptsport]

200 more games 9x9, SAI with additional parameters: 100 games KataGo 10x128-s4588 - SAI (--lambda 0 --mu 0 --nrsymm), (gt 610, 5 sec/move) 55:45 (KataGo b 17:33 and KataGo w 38:12) 100 games KataGo 10x128-s4588 - SAI (--lambda 0 --mu 0 --nrsymm --puct 0), (gt 610, 5 sec/move) 84:16 (KataGo b 39:11 and KataGo w 45:5)

edit: sorry, KataGo both networks s4588

[cryptsport]

new release SAI 0.17.5 100 games KataGo 10x128-s4588 - SAI (--lambda 0 --mu 0 --nrsymm), (gt 610, 5 sec/move) 48:52 (KataGo b 10:40 and KataGo w 38:12)

[cryptsport]

100 games 10 sec/move: KataGo 10x128-s4588 - SAI (--lambda 0 --mu 0 --nrsymm), (gt 610) 51:49 (KataGo b 11:39 and KataGo w 40:10)

[cryptsport]

50 games 5 sec/move: KataGo 10x128-s4588 - ELF v2 (gt 610) 37:13 (KataGo b 16:9 and KataGo w 21:4) 8 games the ELF lost due to the ladder. without these games KataGo - ELF 29:13 (KataGo b 13:9 and KataGo w 16:4)

[Friday9i]

Here is a new version of the 10b, around 20 Elo stronger on average than v2. Here are the result from 200 games per board size at 100 visits, vs g104 10b, 15b and 20b:

10b v3 winrate	6x6	7x7	8x8	9x9	11x11	13x13	19x19
vs 10b g104	66%	93%	85%	76%	71%	81%	85%
vs 15b g104	72%	74%	74%	65%	53%	40%	26%
vs 20b g104	67%	67%	66%	44%	19%	7%	2%

Link to the net and the full folder are there: https://filetransfer.io/data-package/SQA41FYS Hence, on equal time, it is way stronger than g104 20b up to 9x9 sizes, probably slightly strength stronger up to 11x11, weaker above. (But note than @hzyhzy specialized nets by size, for 6x6 to 9x9, are stronger) My 15b nets is now slightly stronger on equal visits, but still weaker on equal time, so I'm not posting it yet. Same for the 30b net (much stronger than 10b at equal time, but not yet on time parity).

[luckyangman]

Excellent work. Not tested with g161?

[kennyfs]

Where is g161?

[luckyangman]

https://drive.google.com/file/d/1caHLk2cuDX9EeZxluk_4nUy9OAvpwEhD/view

[luckyangman]

https://drive.google.com/file/d/1Xa3mpB3Wjen67-nYxETMygeLFrqokPWp/view

[kennyfs]

Is g161 the third run? Where can I find something like "https://d3dndmfyhecmj0.cloudfront.net/" ?

[Friday9i]

And what is g161...?

[luckyangman]

g161-b15c192-s360540160-d118721852.txt.gz

[Friday9i]

Yeah ok it's a net, but what is this net, how is it done, what are the differences, who did it,...?

[luckyangman]

lightvector

[luckyangman]

In the discord on December 7, 2019

[kaorahi]

Do we need some unpublished version of KataGo to use this network? Is it related to this "handicap game style"?? https://lifein19x19.com/viewtopic.php?f=18&t=17183

$ ./katago gtp -model g161-b15c192-s360540160-d118721852.txt.gz -config gtp_my.cfg
KataGo v1.2
terminate called after throwing an instance of 'StringError'
  what():  Error loading or parsing model file /XXX/g161-b15c192-s360540160-d118721852.txt.gz: b15c192-s360540160-d118721852: model found unsupported version 7