search

CSSLab / maia-chess

Maia is a human-like neural network chess engine trained on millions of human games.
https://maiachess.com
GNU General Public License v3.0
964 stars 121 forks source link

Variability in prediction in the same position #25

Closed JotaGreen closed 3 years ago

JotaGreen commented 3 years ago

Do the Maia predictions have some randomness involved? For some positions, it appears the predicted move is not always the same. For example, in the position 5k2/5pp1/7p/2p5/5PPP/1p5R/2P1K3/1r6 w - - 0 33, either isolated or using the move history (1. e4 e5 2. Nf3 Nc6 3. Nc3 Nf6 4. d4 d6 5. Be3 Be7 6. dxe5 dxe5 7. Nxe5 Nxe5 8. f4 Nc6 9. Qe2 O-O 10. Qc4 Be6 11. Qe2 Nd4 12. Qd2 c5 13. e5 Ng4 14. O-O-O Nxe3 15. Qxe3 Qc7 16. Ne4 h6 17. b3 Rfd8 18. h4 a5 19. g3 b5 20. Bh3 Bxh3 21. Rxh3 Nxb3+ 22. Kb2 Rxd1 23. Kxb3 Rb1+ 24. Kc3 b4+ 25. Kd2 a4 26. Nd6 Bxd6 27. exd6 Qxd6+ 28. Ke2 Re8 29. Qxe8+ Qf8 30. Qxf8+ Kxf8 31. g4 b3 32. axb3 axb3), most times the predicted move would be h3b3 but also often it would be c2b3 and rarely h3c3. I am noticing this variability with all ELO networks, using lc0 27.0, in python-chess 1.6.1, macOS 11.4, in a Jupyter notebook. The variability appears larger when I re-start the kernel and run all again, but also happens if I re-run the same cell many times.

reidmcy commented 3 years ago

How are you running the models, Is search disabled? Are you using our model loading code?

JotaGreen commented 3 years ago

I am mostly running the models with a code similar to the below, with search disabled, not using your loading code:

import chess
import chess.engine
import io

maia1100 = chess.engine.SimpleEngine.popen_uci("./lc0")
maia1100.configure({"WeightsFile":"./maia_weights/maia-1100.pb.gz"})

board = chess.Board()
pgn = ("""
1. e4 e5 2. Nf3 Nc6 3. Nc3 Nf6 4. d4 d6 5. Be3 Be7 6. dxe5 dxe5 7. Nxe5 Nxe5 8. f4 Nc6 9. Qe2 O-O 10. Qc4 Be6 11. Qe2 Nd4 12. Qd2 c5 13. e5 Ng4 14. O-O-O Nxe3 15. Qxe3 Qc7 16. Ne4 h6 17. b3 Rfd8 18. h4 a5 19. g3 b5 20. Bh3 Bxh3 21. Rxh3 Nxb3+ 22. Kb2 Rxd1 23. Kxb3 Rb1+ 24. Kc3 b4+ 25. Kd2 a4 26. Nd6 Bxd6 27. exd6 Qxd6+ 28. Ke2 Re8 29. Qxe8+ Qf8 30. Qxf8+ Kxf8 31. g4 b3 32. axb3 axb3
""")
game = chess.pgn.read_game(io.StringIO(pgn))
board = game.board()
for move in game.mainline_moves():
    board.push(move)

resultMaia1100 = maia1100.analyse(board, chess.engine.Limit(nodes=1))
print(resultMaia1100.get("pv")[0])

I could not clearly identify a consistent behaviour, but it appears that the variability in the Maia move prediction sometimes is related to the engine instance: if it is a new instance there is less variability, but if the instance has already evaluated other positions, there is more variability. Also, I have tested running the models in command line, and there is also variability. Copying some command line output examples below, see how the predicted move is different (the moves are from the position I mentioned in the previous comment)

Running just after loading

% ./lc0 --weights=./maia_weights/maia-1100.pb.gz
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [opencl]...
OpenCL, maximum batch size set to 16.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 (May  8 2021 03:14:28)
Platform profile: FULL_PROFILE
Platform name:    Apple
Platform vendor:  Apple
Device ID:      0
Device name:    Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
Device type:    CPU
Device vendor:  Intel
Device driver:  1.1
Device speed:   2400 MHZ
Device cores:   8 CU
Device score:   512
Device ID:      1
Device name:    Intel(R) Iris(TM) Plus Graphics 655
Device type:    GPU
Device vendor:  Intel Inc.
Device driver:  1.2(May  8 2021 07:52:12)
Device speed:   1150 MHZ
Device cores:   48 CU
Device score:   612
Selected platform: Apple
Selected device: Intel(R) Iris(TM) Plus Graphics 655
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning for batch size 16.
Wavefront/Warp size: 8

Max workgroup size: 256
Max workgroup dimensions: 256 256 256
info depth 1 seldepth 1 time 137 nodes 1 score cp -41 tbhits 0 pv h3b3
bestmove h3b3

Running with another position before

% ./lc0 --weights=./maia_weights/maia-1100.pb.gz
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [opencl]...
OpenCL, maximum batch size set to 16.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 (May  8 2021 03:14:28)
Platform profile: FULL_PROFILE
Platform name:    Apple
Platform vendor:  Apple
Device ID:      0
Device name:    Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
Device type:    CPU
Device vendor:  Intel
Device driver:  1.1
Device speed:   2400 MHZ
Device cores:   8 CU
Device score:   512
Device ID:      1
Device name:    Intel(R) Iris(TM) Plus Graphics 655
Device type:    GPU
Device vendor:  Intel Inc.
Device driver:  1.2(May  8 2021 07:52:12)
Device speed:   1150 MHZ
Device cores:   48 CU
Device score:   612
Selected platform: Apple
Selected device: Intel(R) Iris(TM) Plus Graphics 655
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning for batch size 16.
Wavefront/Warp size: 8

Max workgroup size: 256
Max workgroup dimensions: 256 256 256
info depth 1 seldepth 1 time 166 nodes 1 score cp 6 tbhits 0 pv d2d4
bestmove d2d4
ucinewgame
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
info depth 1 seldepth 1 time 3261 nodes 1 score cp -41 tbhits 0 pv c2b3
bestmove c2b3
reidmcy commented 3 years ago

I'm not able to replicate that behaviour. But I don't have access to an opencl backend machine and I tested with a released version of lc0. The d2d4behaviour on the opening board is also suspicious as the models prefer e4. Here is the output from my run with a cudnn backend (blas looks the same):

       _
|   _ | |
|_ |_ |_| v0.27.0 built Jun  8 2021
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [cudnn-auto]...
Switching to [cudnn]...
CUDA Runtime version: 9.1.0
Cudnn version: 8.2.0
Latest version of CUDA supported by the driver: 11.3.0
GPU: NVIDIA Tesla K80
GPU memory: 11.173 GiB
GPU clock frequency: 823.5 MHz
GPU compute capability: 3.7
info depth 1 seldepth 1 time 1211 nodes 1 score cp 6 tbhits 0 pv e2e4
info string b1a3  (34  ) N:       0 (+ 0) (P:  0.10%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00169) (S:  0.04727) (V:  -.----) 
info string g1h3  (161 ) N:       0 (+ 0) (P:  0.19%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00332) (S:  0.04890) (V:  -.----) 
info string a2a3  (204 ) N:       0 (+ 0) (P:  0.34%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00595) (S:  0.05153) (V:  -.----) 
info string f2f3  (346 ) N:       0 (+ 0) (P:  0.35%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00614) (S:  0.05173) (V:  -.----) 
info string h2h3  (400 ) N:       0 (+ 0) (P:  0.44%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00769) (S:  0.05327) (V:  -.----) 
info string a2a4  (207 ) N:       0 (+ 0) (P:  0.53%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00916) (S:  0.05475) (V:  -.----) 
info string h2h4  (403 ) N:       0 (+ 0) (P:  0.64%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01118) (S:  0.05677) (V:  -.----) 
info string g2g4  (378 ) N:       0 (+ 0) (P:  0.73%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01276) (S:  0.05835) (V:  -.----) 
info string b2b4  (234 ) N:       0 (+ 0) (P:  0.76%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01319) (S:  0.05877) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  0.95%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01650) (S:  0.06208) (V:  -.----) 
info string f2f4  (351 ) N:       0 (+ 0) (P:  1.28%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02231) (S:  0.06790) (V:  -.----) 
info string d2d3  (288 ) N:       0 (+ 0) (P:  1.60%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02800) (S:  0.07358) (V:  -.----) 
info string b1c3  (36  ) N:       0 (+ 0) (P:  1.66%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02898) (S:  0.07457) (V:  -.----) 
info string b2b3  (230 ) N:       0 (+ 0) (P:  2.13%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.03717) (S:  0.08276) (V:  -.----) 
info string g2g3  (374 ) N:       0 (+ 0) (P:  2.82%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.04918) (S:  0.09477) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  4.21%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.07349) (S:  0.11908) (V:  -.----) 
info string g1f3  (159 ) N:       0 (+ 0) (P:  4.37%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.07618) (S:  0.12177) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  5.18%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.09035) (S:  0.13593) (V:  -.----) 
info string d2d4  (293 ) N:       0 (+ 0) (P: 21.61%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.37705) (S:  0.42264) (V:  -.----) 
info string e2e4  (322 ) N:       0 (+ 0) (P: 50.12%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.87468) (S:  0.92027) (V:  -.----) 
info string node  (  20) N:       1 (+ 0) (P:  0.00%) (WL:  0.04558) (D: 0.039) (M:  0.0) (Q:  0.04558) (V:  0.0456) 
bestmove e2e4

I also used --verbose-move-stats so we can see the raw policy distribution. Can you try again with the blas or cudnn backend and a released version? Also using --verbose-move-stats would help as that gives more detail.

JotaGreen commented 3 years ago

I believe I am using a release 27.0 version, but it is mislabeled. I have installed from brew and also tested compiling getting from github with git clone -b release/0.27 --recurse-submodules https://github.com/LeelaChessZero/lc0.git and the heading is the same (with the +git.dirty).

Apparently lc0 often causes issues with OpenCl on Mac. I have changed to BLAS and the variability has decreased, but if repeating the same position many times, there is still variability. I am including below some outputs with --verbose-move-stats in case it helps, although this time the move predicted in BLAS are constant.

BLAS

% ./lc0 --weights=./maia_weights/maia-1100.pb.gz --backend=blas --verbose-move-stats
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [blas]...
BLAS vendor: Apple vecLib.
BLAS max batch size is 256.
info depth 1 seldepth 1 time 55 nodes 1 score cp 6 tbhits 0 pv e2e4
info string b1a3  (34  ) N:       0 (+ 0) (P:  0.10%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00169) (S:  0.04727) (V:  -.----) 
info string g1h3  (161 ) N:       0 (+ 0) (P:  0.19%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00332) (S:  0.04890) (V:  -.----) 
info string a2a3  (204 ) N:       0 (+ 0) (P:  0.34%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00595) (S:  0.05153) (V:  -.----) 
info string f2f3  (346 ) N:       0 (+ 0) (P:  0.35%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00614) (S:  0.05173) (V:  -.----) 
info string h2h3  (400 ) N:       0 (+ 0) (P:  0.44%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00769) (S:  0.05327) (V:  -.----) 
info string a2a4  (207 ) N:       0 (+ 0) (P:  0.53%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00916) (S:  0.05475) (V:  -.----) 
info string h2h4  (403 ) N:       0 (+ 0) (P:  0.64%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01118) (S:  0.05676) (V:  -.----) 
info string g2g4  (378 ) N:       0 (+ 0) (P:  0.73%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01276) (S:  0.05835) (V:  -.----) 
info string b2b4  (234 ) N:       0 (+ 0) (P:  0.76%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01319) (S:  0.05877) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  0.95%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.01650) (S:  0.06208) (V:  -.----) 
info string f2f4  (351 ) N:       0 (+ 0) (P:  1.28%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02231) (S:  0.06790) (V:  -.----) 
info string d2d3  (288 ) N:       0 (+ 0) (P:  1.60%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02800) (S:  0.07358) (V:  -.----) 
info string b1c3  (36  ) N:       0 (+ 0) (P:  1.66%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02898) (S:  0.07457) (V:  -.----) 
info string b2b3  (230 ) N:       0 (+ 0) (P:  2.13%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.03717) (S:  0.08276) (V:  -.----) 
info string g2g3  (374 ) N:       0 (+ 0) (P:  2.82%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.04918) (S:  0.09477) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  4.21%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.07349) (S:  0.11908) (V:  -.----) 
info string g1f3  (159 ) N:       0 (+ 0) (P:  4.37%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.07618) (S:  0.12177) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  5.18%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.09035) (S:  0.13593) (V:  -.----) 
info string d2d4  (293 ) N:       0 (+ 0) (P: 21.61%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.37705) (S:  0.42264) (V:  -.----) 
info string e2e4  (322 ) N:       0 (+ 0) (P: 50.12%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.87468) (S:  0.92027) (V:  -.----) 
info string node  (  20) N:       1 (+ 0) (P:  0.00%) (WL:  0.04558) (D: 0.039) (M:  0.0) (Q:  0.04558) (V:  0.0456) 
bestmove e2e4
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
info depth 1 seldepth 1 time 3440 nodes 1 score cp -41 tbhits 0 pv c2b3
info string e2f3  (318 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00151) (S: -0.27687) (V:  -.----) 
info string e2f2  (312 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00160) (S: -0.27679) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  0.14%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00238) (S: -0.27600) (V:  -.----) 
info string h3f3  (638 ) N:       0 (+ 0) (P:  0.15%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00257) (S: -0.27582) (V:  -.----) 
info string h3g3  (639 ) N:       0 (+ 0) (P:  0.16%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00277) (S: -0.27561) (V:  -.----) 
info string h3h2  (632 ) N:       0 (+ 0) (P:  0.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00409) (S: -0.27429) (V:  -.----) 
info string h4h5  (888 ) N:       0 (+ 0) (P:  0.26%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00460) (S: -0.27379) (V:  -.----) 
info string f4f5  (830 ) N:       0 (+ 0) (P:  0.27%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00475) (S: -0.27364) (V:  -.----) 
info string h3e3  (637 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00507) (S: -0.27331) (V:  -.----) 
info string h3h1  (629 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00509) (S: -0.27329) (V:  -.----) 
info string e2d3  (316 ) N:       0 (+ 0) (P:  0.96%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.01668) (S: -0.26171) (V:  -.----) 
info string g4g5  (861 ) N:       0 (+ 0) (P:  1.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02146) (S: -0.25693) (V:  -.----) 
info string h3d3  (636 ) N:       0 (+ 0) (P:  1.54%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02689) (S: -0.25150) (V:  -.----) 
info string h3c3  (635 ) N:       0 (+ 0) (P:  1.66%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02898) (S: -0.24940) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  1.79%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03129) (S: -0.24710) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  2.30%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.04020) (S: -0.23819) (V:  -.----) 
info string e2d2  (311 ) N:       0 (+ 0) (P:  4.02%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.07022) (S: -0.20817) (V:  -.----) 
info string h3b3  (634 ) N:       0 (+ 0) (P:  8.49%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.14816) (S: -0.13023) (V:  -.----) 
info string c2b3  (258 ) N:       0 (+ 0) (P: 76.03%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 1.32672) (S:  1.04834) (V:  -.----) 
info string node  (  19) N:       1 (+ 0) (P:  0.00%) (WL: -0.27839) (D: 0.228) (M:  0.0) (Q: -0.27839) (V: -0.2784) 
bestmove c2b3
% ./lc0 --weights=./maia_weights/maia-1100.pb.gz --backend=blas --verbose-move-stats
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [blas]...
BLAS vendor: Apple vecLib.
BLAS max batch size is 256.
info depth 1 seldepth 1 time 4452 nodes 1 score cp -41 tbhits 0 pv c2b3
info string e2f3  (318 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00151) (S: -0.27687) (V:  -.----) 
info string e2f2  (312 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00160) (S: -0.27679) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  0.14%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00238) (S: -0.27600) (V:  -.----) 
info string h3f3  (638 ) N:       0 (+ 0) (P:  0.15%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00257) (S: -0.27582) (V:  -.----) 
info string h3g3  (639 ) N:       0 (+ 0) (P:  0.16%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00277) (S: -0.27561) (V:  -.----) 
info string h3h2  (632 ) N:       0 (+ 0) (P:  0.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00409) (S: -0.27429) (V:  -.----) 
info string h4h5  (888 ) N:       0 (+ 0) (P:  0.26%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00460) (S: -0.27379) (V:  -.----) 
info string f4f5  (830 ) N:       0 (+ 0) (P:  0.27%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00475) (S: -0.27364) (V:  -.----) 
info string h3e3  (637 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00507) (S: -0.27331) (V:  -.----) 
info string h3h1  (629 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00509) (S: -0.27329) (V:  -.----) 
info string e2d3  (316 ) N:       0 (+ 0) (P:  0.96%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.01668) (S: -0.26171) (V:  -.----) 
info string g4g5  (861 ) N:       0 (+ 0) (P:  1.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02146) (S: -0.25693) (V:  -.----) 
info string h3d3  (636 ) N:       0 (+ 0) (P:  1.54%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02689) (S: -0.25150) (V:  -.----) 
info string h3c3  (635 ) N:       0 (+ 0) (P:  1.66%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02898) (S: -0.24940) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  1.79%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03129) (S: -0.24710) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  2.30%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.04020) (S: -0.23819) (V:  -.----) 
info string e2d2  (311 ) N:       0 (+ 0) (P:  4.02%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.07022) (S: -0.20817) (V:  -.----) 
info string h3b3  (634 ) N:       0 (+ 0) (P:  8.49%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.14816) (S: -0.13023) (V:  -.----) 
info string c2b3  (258 ) N:       0 (+ 0) (P: 76.03%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 1.32672) (S:  1.04834) (V:  -.----) 
info string node  (  19) N:       1 (+ 0) (P:  0.00%) (WL: -0.27839) (D: 0.228) (M:  0.0) (Q: -0.27839) (V: -0.2784) 
bestmove c2b3

OpenCL

% ./lc0 --weights=./maia_weights/maia-1100.pb.gz --verbose-move-stats      
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [opencl]...
OpenCL, maximum batch size set to 16.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 (May  8 2021 03:14:28)
Platform profile: FULL_PROFILE
Platform name:    Apple
Platform vendor:  Apple
Device ID:      0
Device name:    Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
Device type:    CPU
Device vendor:  Intel
Device driver:  1.1
Device speed:   2400 MHZ
Device cores:   8 CU
Device score:   512
Device ID:      1
Device name:    Intel(R) Iris(TM) Plus Graphics 655
Device type:    GPU
Device vendor:  Intel Inc.
Device driver:  1.2(May  8 2021 07:52:12)
Device speed:   1150 MHZ
Device cores:   48 CU
Device score:   612
Selected platform: Apple
Selected device: Intel(R) Iris(TM) Plus Graphics 655
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning for batch size 16.
Wavefront/Warp size: 8

Max workgroup size: 256
Max workgroup dimensions: 256 256 256
info depth 1 seldepth 1 time 188 nodes 1 score cp 6 tbhits 0 pv d2d4
info string g1f3  (159 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string g1h3  (161 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string a2a3  (204 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string a2a4  (207 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string e2e4  (322 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string g2g3  (374 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string g2g4  (378 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00000) (S:  0.04558) (V:  -.----) 
info string b1a3  (34  ) N:       0 (+ 0) (P:  0.32%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.00552) (S:  0.05110) (V:  -.----) 
info string f2f3  (346 ) N:       0 (+ 0) (P:  1.15%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02006) (S:  0.06564) (V:  -.----) 
info string h2h3  (400 ) N:       0 (+ 0) (P:  1.45%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.02524) (S:  0.07083) (V:  -.----) 
info string h2h4  (403 ) N:       0 (+ 0) (P:  2.10%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.03660) (S:  0.08218) (V:  -.----) 
info string b2b4  (234 ) N:       0 (+ 0) (P:  2.47%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.04307) (S:  0.08866) (V:  -.----) 
info string f2f4  (351 ) N:       0 (+ 0) (P:  4.19%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.07307) (S:  0.11865) (V:  -.----) 
info string d2d3  (288 ) N:       0 (+ 0) (P:  5.25%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.09163) (S:  0.13721) (V:  -.----) 
info string b1c3  (36  ) N:       0 (+ 0) (P:  5.44%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.09501) (S:  0.14059) (V:  -.----) 
info string b2b3  (230 ) N:       0 (+ 0) (P:  7.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 0.12217) (S:  0.16775) (V:  -.----) 
info string d2d4  (293 ) N:       0 (+ 0) (P: 70.63%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q:  0.04558) (U: 1.23256) (S:  1.27815) (V:  -.----) 
info string node  (  20) N:       1 (+ 0) (P:  0.00%) (WL:  0.04558) (D: 0.039) (M:  0.0) (Q:  0.04558) (V:  0.0456) 
bestmove d2d4
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
info depth 1 seldepth 1 time 3354 nodes 1 score cp -41 tbhits 0 pv c2b3
info string e2f3  (318 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00151) (S: -0.27687) (V:  -.----) 
info string e2f2  (312 ) N:       0 (+ 0) (P:  0.09%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00160) (S: -0.27679) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  0.14%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00238) (S: -0.27600) (V:  -.----) 
info string h3f3  (638 ) N:       0 (+ 0) (P:  0.15%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00257) (S: -0.27582) (V:  -.----) 
info string h3g3  (639 ) N:       0 (+ 0) (P:  0.16%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00277) (S: -0.27561) (V:  -.----) 
info string h3h2  (632 ) N:       0 (+ 0) (P:  0.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00409) (S: -0.27429) (V:  -.----) 
info string h4h5  (888 ) N:       0 (+ 0) (P:  0.26%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00460) (S: -0.27379) (V:  -.----) 
info string f4f5  (830 ) N:       0 (+ 0) (P:  0.27%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00475) (S: -0.27364) (V:  -.----) 
info string h3e3  (637 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00507) (S: -0.27331) (V:  -.----) 
info string h3h1  (629 ) N:       0 (+ 0) (P:  0.29%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00509) (S: -0.27329) (V:  -.----) 
info string e2d3  (316 ) N:       0 (+ 0) (P:  0.96%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.01668) (S: -0.26171) (V:  -.----) 
info string g4g5  (861 ) N:       0 (+ 0) (P:  1.23%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02146) (S: -0.25693) (V:  -.----) 
info string h3d3  (636 ) N:       0 (+ 0) (P:  1.54%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02689) (S: -0.25150) (V:  -.----) 
info string h3c3  (635 ) N:       0 (+ 0) (P:  1.66%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02898) (S: -0.24940) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  1.79%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03129) (S: -0.24710) (V:  -.----) 
info string c2c3  (259 ) N:       0 (+ 0) (P:  2.30%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.04020) (S: -0.23819) (V:  -.----) 
info string e2d2  (311 ) N:       0 (+ 0) (P:  4.02%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.07022) (S: -0.20817) (V:  -.----) 
info string h3b3  (634 ) N:       0 (+ 0) (P:  8.49%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.14816) (S: -0.13023) (V:  -.----) 
info string c2b3  (258 ) N:       0 (+ 0) (P: 76.03%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 1.32672) (S:  1.04834) (V:  -.----) 
info string node  (  19) N:       1 (+ 0) (P:  0.00%) (WL: -0.27839) (D: 0.228) (M:  0.0) (Q: -0.27839) (V: -0.2784) 
bestmove c2b3
% ./lc0 --weights=./maia_weights/maia-1100.pb.gz --verbose-move-stats
       _
|   _ | |
|_ |_ |_| v0.27.0+git.dirty built Feb 21 2021
position startpos moves e2e4 e7e5 g1f3 b8c6 b1c3 g8f6 d2d4 d7d6 c1e3 f8e7 d4e5 d6e5 f3e5 c6e5 f2f4 e5c6 d1e2 e8g8 e2c4 c8e6 c4e2 c6d4 e2d2 c7c5 e4e5 f6g4 e1c1 g4e3 d2e3 d8c7 c3e4 h7h6 b2b3 f8d8 h2h4 a7a5 g2g3 b7b5 f1h3 e6h3 h1h3 d4b3 c1b2 d8d1 b2b3 d1b1 b3c3 b5b4 c3d2 a5a4 e4d6 e7d6 e5d6 c7d6 d2e2 a8e8 e3e8 d6f8 e8f8 g8f8 g3g4 b4b3 a2b3 a4b3
go nodes 1
Loading weights file from: ./maia_weights/maia-1100.pb.gz
Creating backend [opencl]...
OpenCL, maximum batch size set to 16.
Initializing OpenCL.
Detected 1 OpenCL platforms.
Platform version: OpenCL 1.2 (May  8 2021 03:14:28)
Platform profile: FULL_PROFILE
Platform name:    Apple
Platform vendor:  Apple
Device ID:      0
Device name:    Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
Device type:    CPU
Device vendor:  Intel
Device driver:  1.1
Device speed:   2400 MHZ
Device cores:   8 CU
Device score:   512
Device ID:      1
Device name:    Intel(R) Iris(TM) Plus Graphics 655
Device type:    GPU
Device vendor:  Intel Inc.
Device driver:  1.2(May  8 2021 07:52:12)
Device speed:   1150 MHZ
Device cores:   48 CU
Device score:   612
Selected platform: Apple
Selected device: Intel(R) Iris(TM) Plus Graphics 655
with OpenCL 1.2 capability.
Loaded existing SGEMM tuning for batch size 16.
Wavefront/Warp size: 8

Max workgroup size: 256
Max workgroup dimensions: 256 256 256
info depth 1 seldepth 1 time 200 nodes 1 score cp -41 tbhits 0 pv h3b3
info string c2c3  (259 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string c2c4  (264 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string c2b3  (258 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string e2d2  (311 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string e2f2  (312 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string e2d3  (316 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string e2e3  (317 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string e2f3  (318 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string g4g5  (861 ) N:       0 (+ 0) (P:  0.00%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.00000) (S: -0.27839) (V:  -.----) 
info string h3f3  (638 ) N:       0 (+ 0) (P:  1.10%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.01922) (S: -0.25917) (V:  -.----) 
info string h3g3  (639 ) N:       0 (+ 0) (P:  1.19%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.02078) (S: -0.25760) (V:  -.----) 
info string h3h2  (632 ) N:       0 (+ 0) (P:  1.76%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03072) (S: -0.24767) (V:  -.----) 
info string h4h5  (888 ) N:       0 (+ 0) (P:  1.97%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03439) (S: -0.24400) (V:  -.----) 
info string f4f5  (830 ) N:       0 (+ 0) (P:  2.03%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03550) (S: -0.24289) (V:  -.----) 
info string h3e3  (637 ) N:       0 (+ 0) (P:  2.17%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03795) (S: -0.24044) (V:  -.----) 
info string h3h1  (629 ) N:       0 (+ 0) (P:  2.18%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.03808) (S: -0.24031) (V:  -.----) 
info string h3d3  (636 ) N:       0 (+ 0) (P: 11.58%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.20211) (S: -0.07628) (V:  -.----) 
info string h3c3  (635 ) N:       0 (+ 0) (P: 12.48%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 0.21782) (S: -0.06057) (V:  -.----) 
info string h3b3  (634 ) N:       0 (+ 0) (P: 63.53%) (WL:  -.-----) (D: -.---) (M:  -.-) (Q: -0.27839) (U: 1.10858) (S:  0.83020) (V:  -.----) 
info string node  (  19) N:       1 (+ 0) (P:  0.00%) (WL: -0.27839) (D: 0.228) (M:  0.0) (Q: -0.27839) (V: -0.2784) 
bestmove h3b3
reidmcy commented 3 years ago

It looks like OpenCL is causing the issues. That policy distribution doesn't look like what the models produce, since they always have some small value (scaled by the softmax so the absolute value is variable) for every legal move.

The variability you're seeing with BLAS is it from a fresh start? Or from repeated runs with the same position? If it's the latter are you disabling caching?

We did all our testing with 0.18.1 using BLAS, https://github.com/CSSLab/lc0 has the exact version we used. The models should be deterministic as they're just neurel networks with no sources of entropy.

JotaGreen commented 3 years ago

You got it right, the variability I was seeing with BLAS only happened on repeated runs in the same position. I disabled caching and now everything seems to be working fine. Perhaps it could be added to the Readme on the instructions to run locally that the backend should be set to BLAS and that NNCacheSize should be set to 0, to avoid that other people have similar issues in the future.

For context, I came across this issue when trying to develop an UCI compatible interface to Maia (https://github.com/JotaGreen/MaiaChessUCI). I am finding very interesting and instructional to analyze my games with it.

Thanks a lot for the help on this issue, as well as for doing this great project and sharing it.

reidmcy commented 3 years ago

Great to hear. The BLAS backend is the most stable across machines. But I use Cuda on most of my stuff and was only able to find a couple positions out of millions where they disagree. And that only happens when tow positions have almost identical policy's so it's probably just floating point differences.

As for the cache, we don't disable it and instead restart the models between games. So I'm hesitant to recommend that as the caching should only change things if you're running the model for a long time and I haven't tested disabling caching.

Thanks for the link to the project It's great to see other people building on our work.