Non-reproducible search after ucinewgame

[vondele]

I was expecting the (single threaded) search to be reproducible after a 'ucinewgame'. It is also what the comment in the code suggests:

/// Search::clear() resets search state to zero, to obtain reproducible results

However, I noticed that this is not the case, and sometimes analysing games different bestmoves are suggested depending on the history of games played.

To reproduce, the following expect script can be used (or typed at the cli)

 set timeout 10
 spawn ./stockfish
 send "uci\n"
 expect "uciok"

 send "ucinewgame\n"
 send "position startpos\n"
 send "go nodes 8000\n"
 expect "bestmove"

 send "position startpos e2e4 e7e6\n"
 send "go nodes 8000\n"
 expect "bestmove"

 send "ucinewgame\n"
 send "position startpos\n"
 send "go nodes 8000\n"
 expect "bestmove"

 send "position startpos e2e4 e7e6\n"
 send "go nodes 8000\n"
 expect "bestmove"

 send "quit\n"
 expect eof

which results in

spawn ./stockfish
uci
Stockfish 8 64 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
id name Stockfish 8 64
id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott

option name Debug Log File type string default 
option name Contempt type spin default 0 min -100 max 100
option name Threads type spin default 1 min 1 max 128
option name Hash type spin default 16 min 1 max 1048576
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 89 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 6 min 0 max 6
uciok
ucinewgame
position startpos
go nodes 8000
info depth 1 seldepth 1 multipv 1 score cp 90 nodes 20 nps 20000 tbhits 0 time 1 pv e2e4
info depth 2 seldepth 2 multipv 1 score cp 93 nodes 47 nps 47000 tbhits 0 time 1 pv e2e4 b7b6
info depth 3 seldepth 3 multipv 1 score cp 119 nodes 133 nps 66500 tbhits 0 time 2 pv d2d4 d7d6 e2e4
info depth 4 seldepth 4 multipv 1 score cp 15 nodes 591 nps 197000 tbhits 0 time 3 pv g1f3 d7d5 d2d4 e7e6
info depth 5 seldepth 5 multipv 1 score cp 48 nodes 997 nps 332333 tbhits 0 time 3 pv g1f3 d7d5 d2d4 e7e6 e2e3
info depth 6 seldepth 6 multipv 1 score cp 27 nodes 2111 nps 527750 tbhits 0 time 4 pv e2e4 b8c6 d2d4 d7d5 e4e5 e7e6
info depth 7 seldepth 7 multipv 1 score cp 42 nodes 4986 nps 831000 tbhits 0 time 6 pv g1f3 d7d5 e2e3 g8f6 b1c3 e7e6 d2d4
info depth 8 seldepth 9 multipv 1 score cp 27 nodes 10605 nps 707000 tbhits 0 time 15 pv e2e4 d7d5 e4d5 d8d5 g1f3 b8c6 b1c3 d5e6 d1e2
info depth 9 seldepth 14 multipv 1 score cp 41 nodes 14985 nps 788684 tbhits 0 time 19 pv e2e4
bestmove e2e4 ponder d7d5
position startpos e2e4 e7e6
go nodes 8000
info depth 1 seldepth 1 multipv 1 score cp 52 nodes 34 nps 34000 tbhits 0 time 1 pv e2e4 d7d5 e4d5
info depth 2 seldepth 2 multipv 1 score cp 52 nodes 62 nps 62000 tbhits 0 time 1 pv e2e4 d7d5
info depth 3 seldepth 3 multipv 1 score cp 52 nodes 95 nps 95000 tbhits 0 time 1 pv e2e4 d7d5 e4d5
info depth 4 seldepth 4 multipv 1 score cp 52 nodes 139 nps 139000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6
info depth 5 seldepth 5 multipv 1 score cp 52 nodes 254 nps 254000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6 g1f3
info depth 6 seldepth 6 multipv 1 score cp 52 nodes 416 nps 416000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6 g1f3 f6d5
info depth 7 seldepth 7 multipv 1 score cp 44 nodes 818 nps 818000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6 d1e2 c7c6 d5c6
info depth 8 seldepth 8 multipv 1 score cp 44 nodes 1195 nps 597500 tbhits 0 time 2 pv e2e4 d7d5 e4d5 g8f6 d1e2 c7c6 d5c6 b8c6
info depth 9 seldepth 12 multipv 1 score cp 29 nodes 11095 nps 1232777 tbhits 0 time 9 pv e2e4 d7d5 e4d5 g8f6 d2d4 c7c6 b1c3 f6d5 g1f3 c8g4
info depth 10 seldepth 14 multipv 1 score cp 29 nodes 14767 nps 1230583 tbhits 0 time 12 pv e2e4 d7d5 e4d5 g8f6 d2d4 c7c6 b1c3 f6d5 g1f3 c8g4
bestmove e2e4 ponder d7d5
ucinewgame
position startpos
go nodes 8000
info depth 1 seldepth 1 multipv 1 score cp 90 nodes 20 nps 20000 tbhits 0 time 1 pv e2e4
info depth 2 seldepth 2 multipv 1 score cp 93 nodes 47 nps 23500 tbhits 0 time 2 pv e2e4 b7b6
info depth 3 seldepth 3 multipv 1 score cp 119 nodes 133 nps 66500 tbhits 0 time 2 pv d2d4 d7d6 e2e4
info depth 4 seldepth 4 multipv 1 score cp 15 nodes 591 nps 295500 tbhits 0 time 2 pv g1f3 d7d5 d2d4 e7e6
info depth 5 seldepth 5 multipv 1 score cp 48 nodes 997 nps 332333 tbhits 0 time 3 pv g1f3 d7d5 d2d4 e7e6 e2e3
info depth 6 seldepth 6 multipv 1 score cp 27 nodes 2111 nps 527750 tbhits 0 time 4 pv e2e4 b8c6 d2d4 d7d5 e4e5 e7e6
info depth 7 seldepth 7 multipv 1 score cp 42 nodes 4986 nps 712285 tbhits 0 time 7 pv g1f3 d7d5 e2e3 g8f6 b1c3 e7e6 d2d4
info depth 8 seldepth 9 multipv 1 score cp 27 nodes 10605 nps 1060500 tbhits 0 time 10 pv e2e4 d7d5 e4d5 d8d5 g1f3 b8c6 b1c3 d5e6 d1e2
info depth 9 seldepth 14 multipv 1 score cp 41 nodes 14982 nps 1152461 tbhits 0 time 13 pv e2e4
bestmove e2e4 ponder d7d5
position startpos e2e4 e7e6
go nodes 8000
info depth 1 seldepth 1 multipv 1 score cp 52 nodes 34 nps 34000 tbhits 0 time 1 pv e2e4 d7d5 e4d5
info depth 2 seldepth 2 multipv 1 score cp 52 nodes 62 nps 62000 tbhits 0 time 1 pv e2e4 d7d5
info depth 3 seldepth 3 multipv 1 score cp 52 nodes 95 nps 95000 tbhits 0 time 1 pv e2e4 d7d5 e4d5
info depth 4 seldepth 4 multipv 1 score cp 52 nodes 139 nps 139000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6
info depth 5 seldepth 5 multipv 1 score cp 52 nodes 254 nps 254000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6 g1f3
info depth 6 seldepth 6 multipv 1 score cp 52 nodes 416 nps 416000 tbhits 0 time 1 pv e2e4 d7d5 e4d5 g8f6 g1f3 f6d5
info depth 7 seldepth 7 multipv 1 score cp 44 nodes 818 nps 409000 tbhits 0 time 2 pv e2e4 d7d5 e4d5 g8f6 d1e2 c7c6 d5c6
info depth 8 seldepth 8 multipv 1 score cp 44 nodes 1195 nps 597500 tbhits 0 time 2 pv e2e4 d7d5 e4d5 g8f6 d1e2 c7c6 d5c6 b8c6
info depth 9 seldepth 12 multipv 1 score cp 29 nodes 11098 nps 1387250 tbhits 0 time 8 pv e2e4 d7d5 e4d5 g8f6 d2d4 c7c6 b1c3 f6d5 g1f3 c8g4
info depth 10 seldepth 14 multipv 1 score cp 29 nodes 14770 nps 1342727 tbhits 0 time 11 pv e2e4 d7d5 e4d5 g8f6 d2d4 c7c6 b1c3 f6d5 g1f3 c8g4
bestmove e2e4 ponder d7d5
quit

where the first analysis of the start pos leads to 'nodes 14985' being searched and the second to 'nodes 14982'. In this case, it is a small difference, different bestmoves is not so nice.

I'm wondering if anybody can figure out which piece of state is not reset between the two games, and what should be added to the uicnewgame reset.

[lantonov]

I can't say why constant nodes is not reproducible but in my experience constant depth IS reproducible. Just play 1000 games in, e.g. cutechess-cli between the same SF build and you will see that all games are exactly the same, even if they contain 150+ moves. This pertains to a lower extent to games between different builds (some differing moves at the end, especially in longer games) as well as between Stockfish and BrainFish, or Stockfish-CFish, or Stockfish- pedantFish. From your listing it is seen that at given depth (e.g., depth 9), all runs have the same pv (some only the first move)

[stockfishdeveloper]

@lantonov I've played my share of Stockfish games but I don't think I've ever seen two identical versions of Stockfish play the exact same game. I guess it could have something to do with the time control?

[vondele]

@lantonov yes, I can confirm that constant depth is reproducible.

@stockfishdeveloper if the search is based on time, it will not give identical games, however, on depth or on nodes, it should.

[stockfishdeveloper]

Yes, this is what makes two identical builds have the same bench.

[vondele]

@stockfishdeveloper actually the issue is still slightly different. Things are reproducible also with nodes if the games are played in the same order, it is when the order changes, or as in the example above, played twice, that it is not giving the same answer.

[stockfishdeveloper]

Well, obviously some table is not being cleared when we get the "ucinewgame" command. As far as I know the UCI protocol calls for literally a complete clean of everything to start a new game(or position) fresh.

[lucasart]

git bissect

[vondele]

@lucasart I tried SF8 SF7 SF6 and those commits suggested by blame on search::clear, but couldn't find any good commit.

[mcostalba]

Fixed by #862