Upgrade KataGo from AlphaZero-like to MuZero-like

[GD-fix]

What about to upgrade KataGo for incorporating learned model into the training procedure like in MuZero(https://arxiv.org/pdf/1911.08265.pdf)?

[lightvector]

Even if it were free to write all the code and test everything, probably a not a great idea, because by my understanding:

• Memory requirements increase massively because of the need to store the learned representations of the net at each search tree node. Many users on high-end GPUs may suddenly become unable to search very long before running out of RAM.
• MuZero does not scale indefinitely. You can see in Deepmind's own paper that AlphaZero overtakes it with very long time controls, due to the gradual compounding inaccuracy of its internal learned representations.
• Search tree reuse and NNCache pose an interesting question. If you implement the search as stated in their paper, it's not clear if you can do any sort of tree reuse or caching of the NN across positions, or how that is handled. There is a chance you just lose some of the benefits of them - and unfortunately as the case with many of Deepmind's papers, the paper seems to be ambiguous on how such "engineering details" were dealt with, or whether the comparison was a "fair" comparison (i.e. if indeed MuZero had to give up tree reuse, did AlphaZero get to use tree reuse?).

[GD-fix]

OK. I supposed that not because of smaller learning procedure, but generally because of playing more efficiency exactly with short time controls (i.e. with normal time controls on PC's without high-end GPUs, i.e. on most users PC's), after all: "AlphaGo Zero and AlphaZero use knowledge of the rules of the game in three places: (1) state transitions in the search tree, (2) actions available at each node of the search tree, (3) episode termination within the search tree. In MuZero, all of these have been replaced with the use of a single implicit model learned by a neural network"... For example, what You think about running KataGo on PC with 4 core CPU, 16 GB RAM and GeForce 5500GT?

[GD-fix]

P.S. Will it work opencl binary version at all, or it's better to compile from source code via -DUSE_BACKEND=EIGEN for cmake?

[lightvector]

Still not very interesting. Depending on how tree reuse works out (again, the paper is ambiguous), it's possible that you're losing efficiency even on weaker hardware, rather than gaining it. MuZero mostly seems to me only interesting because of generality - but there's no particular reason to want to offload learning the rules to the neural net when you actually know the rules already and can implement them simply and efficiently.

As for your PC - sounds like you should just try it. Ask for help in the help channel here https://discord.gg/bqkZAz3 if you run into problems, and feel free to open a new issue here if beyond merely GPU compatibility or a problem with your hardware, you discover an actual bug within KataGo.

I'm a bit confused why people keep asking me if they should use the Eigen code when I explicitly said that it's unusable and not ready for use. Either compile yourself from kaorahi's branch (which does contain some excellent work making it good and usable, unlike the current official tip of this repo) or be patient for that branch's code to make it into the next release.

[GD-fix]

I asked, because, for example, LeelaZero developer said, that with such videocards opencl is unusable (has no reason) and it's better to use CPU only version... Thanks for kaorahi's branch: I shall try it, if opencl Linux binary version will not run on my PC!

[GD-fix]

Running of katago-v1.5.0-opencl-linux-x64 get error: ./katago benchmark -model ../g170-b6c96-s175395328-d26788732.bin.gz 2020-08-09 09:13:14+0400: Loading model and initializing benchmark... 2020-08-09 09:13:14+0400: nnRandSeed0 = 4085051146609872344 2020-08-09 09:13:14+0400: After dedups: nnModelFile0 = ../g170-b6c96-s175395328-d26788732.bin.gz useFP16 auto useNHWC auto terminate called after throwing an instance of 'StringError' what(): OpenCL error at /home/dwugcloud/data/kata/cpp/neuralnet/openclhelpers.cpp, func err, line 263, error CL_PLATFORM_NOT_FOUND_KHR

System (OpenSUSE) has: libOpenCL1 - OpenCL ICD Bindings (v.2.2.11-2.4) /usr/lib64/libOpenCL.so.1 /usr/lib64/libOpenCL.so.1.0.0 /usr/share/doc/packages/libOpenCL1 /usr/share/doc/packages/libOpenCL1/README opencl-headers - OpenCL (Open Computing Language) headers (v.2.2+git.20170617-1.3) /usr/include/CL /usr/include/CL/cl.h /usr/include/CL/cl_d3d10.h /usr/include/CL/cl_d3d11.h /usr/include/CL/cl_dx9_media_sharing.h /usr/include/CL/cl_dx9_media_sharing_intel.h /usr/include/CL/cl_egl.h /usr/include/CL/cl_ext.h /usr/include/CL/cl_ext_intel.h /usr/include/CL/cl_gl.h /usr/include/CL/cl_gl_ext.h /usr/include/CL/cl_platform.h /usr/include/CL/cl_va_api_media_sharing_intel.h /usr/include/CL/opencl.h

What is the "/home/dwugcloud/data/kata/cpp/neuralnet/openclhelpers.cpp"? What packets of these one (https://software.opensuse.org/search?utf8=%E2%9C%93&baseproject=openSUSE%3AFactory&q=opencl) I must install, or it's better to try compile myself from kaorahi's branch?

[lightvector]

Do you have a decently new GPU (from the last few years) on that machine and up-to-date drivers for that GPU? OpenCL still needs an actual device like a GPU to work, it can also work with CPU but depending on the system you may explicitly have to go out of your way to install some appropriate Intel framework or something. All that the error message is saying is that you don't have an OpenCL platform/device.

kaorahi's work has been recently been merged in to master and further optimized and improved. Although there are troubles compiling on Windows that I have yet to debug, Linux should be good. Try compiling the Eigen backend yourself from the tip of master on this repo.

[lightvector]

And maybe don't keep using a thread titled "Upgrade KataGo from AlphaZero-like to MuZero-like" to ask OpenCL debugging questions? :)

I would have liked to leave this issue open as info for anyone wondering why MuZero may or may not be interesting for computer Go, but anyone reading through this thread wanting to know will now also have to skim through a whole bunch of irrelevant stuff before realizing that only the first few posts have anything to do with the topic.

[GD-fix]

GeForce 5500GT is old videocard, so GPU is old too (first real version of CUDA didn't work with last working with this card Nvidia firmware driver: now it's used open source driver from OS distribution). I think, appropriate Intel framework or something will not be helpfull for OpenCL with Core 2 quad CPU family too...

OK. I'll try to compile from master branch with the Eigen backend and create new issue, if there will be some troubles.

[lightvector]

Cool. You might also find it useful to ask for help in a help channel from the LZ discord in https://discord.gg/bqkZAz3 . If it's just an issue specific to a hardware and system ("help me install things, why isn't it finding my GPU"), it's often easier to help someone debug if you can chat back and forth in real time, you post an error, they ask you for this or that detail about your system, you respond, etc. Slightly fewer people actively read this Github, so it's better reserved for if you think there is something that needs to be fixed or can be improved in KataGo itself.

[GD-fix]

I install (compile) and run Go on another PC (not here) without any net connection. So, I think, chat will not be usefull in such case...

I tried to compile master branch version and got error not specific to a hardware and system, I think. So, I created new issue: https://github.com/lightvector/KataGo/issues/305

[lightvector]

Anyways, MuZero seems not that interesting as a followup direction, and everything else is off-topic debugging at this point, so closing this thread now.