search

official-stockfish / Stockfish

A free and strong UCI chess engine
https://stockfishchess.org/
GNU General Public License v3.0
11.56k stars 2.27k forks source link

SF needs to stop suiciding! #2620

Closed adentong closed 4 years ago

adentong commented 4 years ago

I've seen this many, many, many, many, many times. When SF seems to be losing and yet the eval is sort of flatlined, it would randomly push a pawn and suicide instead of playing for 50 moves. Can we make it so that SF plays for 50 moves as much as possible and not suicide? Even if this doesn't get merged into master it's useful to have in a special tournament build.

adentong commented 4 years ago

Happened yet again in CCC just now. Threw away the fortress that leela was completely clueless about.

linrock commented 4 years ago

Two examples of Stockfish recently throwing a fortress away vs Leela:

98... b4 by Stockfish 310320 64 BMI2

Move 87. b4 by Stockfish 20200407DC

MichaelB7 commented 4 years ago

It’s helpful to post fen. Thx.

linrock commented 4 years ago

i've edited my comment to include the FEN right before Stockfish's move for both games

vondele commented 4 years ago

Maybe, it would be interesting to see if bae019b53e5c2bfcf0d69b4ebfc52b4f4de762eb influenced that behavior close to the 50moves rule.

adentong commented 4 years ago

Just happened again in superfinals game 26, in which sf needlessly made pawn moves that made defense much more difficult.

Alayan-stk-2 commented 4 years ago

@vondele Might be responsible for the otherwise inexplicable play in https://www.chess.com/computer-chess-championship#event=ccc13-finals&game=63

vondele commented 4 years ago

@Alayan-stk-2 could you try to investigate that a bit carefully, i.e. see if the behavior really changes with this patch? @joergoster, it that something you could look into?

joergoster commented 4 years ago

@adentong Game 26 of TCEC Superfinal was very likely already lost anyways.

@vondele I will take a look.

joergoster commented 4 years ago

It looks like the blunder in game 63 was not 161. .. Kf5-e4, but the following 162. .. Rc3-c2+. Instead 162. .. Be6-d7 seems to hold.

7 Threads, 2 GB Hash, 6-man syzygy bases, 3 min, multipv=10:

info depth 29 seldepth 37 multipv 1 score cp -75 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 d1c2 e4f3 c2b3 f3e4 b3c3 e4f5 c3b4 f5e6 b4c5 d7c8 c5b6 e6e7 b6c6 e7e6 e5g3 c8a6 g3h4 a6c8 h4e1 c8a6 e1a5 a6c8 a5d2 e6e7 d2f4 e7e6 c6b6 e6e7 b6c5 e7e6 info depth 29 seldepth 57 multipv 2 score cp -2042 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3c2 f2g3 e6d7 g3h4 c2c6 h4g5 e4d3 f8d8 d7h3 d8d5 d3e4 d5d8 h3f5 e5d6 f5h3 d6g3 e4f3 d4d5 c6c1 g5h4 c1h1 d5d6 h3g4 h4g5 h1h5 g5g6 h5c5 d6d7 g4h5 g6f6 c5c6 f6e7 c6g6 d8f8 f3g3 c7c8q g3g2 c8c2 g2g3 c2c7 g3h3 c7c3 h3h4 d7d8q info depth 29 seldepth 44 multipv 3 score cp -2183 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv e4d3 f2g3 info depth 28 seldepth 37 multipv 4 score cp -5446 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3c4 f2e2 c4c2 e2d1 c2g2 c7c8q e6c8 f8c8 e4d3 d1c1 g2g6 c8c5 d3e4 c1d2 g6g2 d2c3 g2g1 c5a5 g1g6 a5a8 e4e3 a8c8 g6b6 c8c5 e3e4 c3d2 b6b2 c5c2 b2b7 c2c8 b7a7 info depth 28 seldepth 35 multipv 5 score cp -5446 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3c6 f2e2 c6c2 e2d1 c2g2 c7c8q e6c8 f8c8 e4d3 d1c1 g2g6 c8c5 d3e4 c1d2 g6g2 d2c3 g2g1 c5a5 g1g6 a5a8 e4e3 a8c8 g6b6 c8c5 e3e4 c3d2 b6b2 c5c2 b2b7 c2c8 b7a7 info depth 28 seldepth 33 multipv 6 score cp -5446 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3c1 f2e2 c1c2 e2d1 c2g2 c7c8q e6c8 f8c8 e4d3 d1c1 g2g6 c8c5 d3e4 c1d2 g6g2 d2c3 g2g1 c5a5 g1g6 a5a8 e4e3 a8c8 g6b6 c8c5 e3e4 c3d2 b6b2 c5c2 b2b7 c2c8 b7a7 info depth 28 seldepth 37 multipv 7 score cp -5827 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3b3 c7c8q e6c8 f8c8 b3b2 f2g3 b2b3 g3h4 b3f3 h4g5 f3f5 g5g6 f5f2 c8h8 e4d3 e5f6 f2f1 g6f7 f1a1 f7e6 a1a5 e6f5 d3e2 f6e5 e2f2 h8d8 f2e3 f5g4 e3e4 d8f8 e4d3 info depth 28 seldepth 41 multipv 8 score cp -5827 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3h3 f2e2 h3e3 e2d2 e3d3 d2c2 e6d7 c7c8q d7c8 f8c8 d3a3 c2d2 a3a6 c8h8 a6a2 d2c3 a2a3 c3b4 a3a6 h8h4 e4d3 b4c5 a6a5 c5d6 d3c3 h4h8 c3d3 h8c8 d3e4 c8c1 a5a6 d6c5 a6a5 c5b4 a5a2 info depth 28 seldepth 34 multipv 9 score cp -5845 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv e6f5 f2e2 c3c2 e2d1 c2c3 d1d2 c3d3 d2c2 f5d7 f8d8 d7f5 d8e8 d3h3 c7c8q f5c8 e5g7 e4f5 e8c8 h3h7 c8f8 f5e4 f8e8 e4f5 g7e5 h7b7 e5g3 f5g4 e8d8 b7b5 g3d6 g4f3 c2c3 b5b7 info depth 28 seldepth 30 multipv 10 score cp -14896 nodes 1845617389 nps 10253372 hashfull 973 tbhits 10812407 time 180001 pv c3e3 f8e8 e3f3 f2e2 e6d7 c7c8q d7c8 e5d6 e4f5 e2f3 c8a6 e8e5 f5g6 f3e3 a6c4 e3d2 g6g7 d2c3 g7h7 c3b4 h7g7 b4c5 c4a2 e5d5 g7f6 bestmove e6d7 ponder f2e2

Could someone please confirm? This is the position command taken from the logfile:

position startpos moves e2e4 c7c6 d2d4 d7d5 b1c3 d5e4 c3e4 b8d7 f1d3 g8f6 e4g5 d8c7 g1f3 h7h6 g5e6 c7d6 e6f8 d7f8 c2c3 c8g4 e1g1 d6d5 d3e2 f8g6 h2h3 g4f5 f1e1 e8g8 e2f1 a8d8 d1e2 f8e8 c3c4 d5a5 b2b3 e7e6 c1b2 g6e7 e2e3 f5h7 e3f4 a5f5 f4c7 d8d7 c7h2 e7g6 e1e3 f5a5 a2a4 a5c7 h2c7 d7c7 b3b4 g6f8 f3d2 e8d8 b4b5 f8d7 d2b3 h7c2 b3a5 d8c8 b2a3 f6e8 a3b4 g7g5 f2f3 c2g6 e3e1 g8g7 e1d1 d7f6 b4e1 f6h5 d1d2 c8a8 d2b2 e8f6 g2g3 g7h7 c4c5 f6d5 a5c4 f7f6 b5c6 b7c6 a4a5 h5g7 a5a6 g7f5 e1f2 h7g8 g3g4 f5g7 b2b7 c7f7 c4a5 a8c8 a1a3 h6h5 a3b3 f6f5 b7f7 g6f7 b3b7 c8c7 a5c4 h5g4 h3g4 g7e8 c4e5 e8f6 f2e1 f7e8 e1a5 c7c8 g4f5 c8a8 f5e6 d5f4 g1f2 f6d5 a5d2 f4e6 f1c4 e6d8 b7b2 d8f7 c4d5 c6d5 e5f7 g8f7 d2g5 e8c6 g5f4 a8d8 f4e5 d8e8 f2e3 f7e6 b2h2 c6b5 h2b2 b5a6 b2a2 a6b5 a2a7 e8e7 a7a5 e7b7 e3f2 e6d7 a5a3 b5c4 a3a8 c4d3 a8a3 d3c4 f3f4 c4b5 f2g3 d7e6 g3g4 b7f7 g4g5 f7f5 g5g4 f5f8 a3a7 f8f5 a7a5 b5e2 g4g3 f5f8 g3f2 e2d3 a5a3 d3b5 a3b3 b5d7 b3b6 e6f5 f2f3 d7e6 c5c6 f8a8 c6c7 a8a3 f3e2 a3a2 e2d3 a2a3 d3c2 a3a4 c2b3 a4c4 b6b8 f5e4 b8h8 e6d7 h8d8 d7h3 d8h8 h3d7 h8h7 d7a4 b3a3 a4b5 h7h8 b5d7 h8h7 d7e6 a3b3 e6g4 h7g7 g4e6 g7g6 e6d7 g6g3 d7e6 b3b2 e4f5 g3g7 f5e4 b2b3 e6f5 g7g5 f5d7 g5g1 d7e6 g1g2 e6h3 g2g1 h3e6 g1g3 e6f5 b3b2 f5d7 g3a3 c4b4 b2c3 b4c4 c3d2 d7e6 f4f5 e4f5 a3a6 e6d7 a6a8 f5e6 a8h8 e6f7 d2e3 c4c3 e3f4 f7e7 h8h7 e7e6 f4g5 c3c1 g5g4 c1f1 g4g3 f1c1 g3f2 c1c3 f2e2 d7c8 h7h8 c8d7 e2d2 c3c4 h8h7 d7c8 d2e3 c4c1 h7h8 c8d7 e3f4 c1f1 f4g3 f1c1 g3f3 e6f5 h8h7 c1c3 f3e2 f5e6 e2f2 d7c8 h7h8 c8d7 f2e2 c3c2 e2f3 c2c1 h8h7 c1c2 h7g7 c2c1 f3e3 c1c2 g7g1 e6f5 g1g7 d7e6 g7h7 f5g5 h7h8 c2c3 e3d2 c3c4 h8e8 g5f5 d2e3 c4c3 e3f2 e6d7 e8d8 d7e6 d8h8 f5e4 h8f8

joergoster commented 4 years ago

And this is the relevant part of the logfile. Thanks to @Alayan-stk-2 for providing it!

3388782 >Stockfish(1): position startpos moves e2e4 c7c6 d2d4 d7d5 b1c3 d5e4 c3e4 b8d7 f1d3 g8f6 e4g5 d8c7 g1f3 h7h6 g5e6 c7d6 e6f8 d7f8 c2c3 c8g4 e1g1 d6d5 d3e2 f8g6 h2h3 g4f5 f1e1 e8g8 e2f1 a8d8 d1e2 f8e8 c3c4 d5a5 b2b3 e7e6 c1b2 g6e7 e2e3 f5h7 e3f4 a5f5 f4c7 d8d7 c7h2 e7g6 e1e3 f5a5 a2a4 a5c7 h2c7 d7c7 b3b4 g6f8 f3d2 e8d8 b4b5 f8d7 d2b3 h7c2 b3a5 d8c8 b2a3 f6e8 a3b4 g7g5 f2f3 c2g6 e3e1 g8g7 e1d1 d7f6 b4e1 f6h5 d1d2 c8a8 d2b2 e8f6 g2g3 g7h7 c4c5 f6d5 a5c4 f7f6 b5c6 b7c6 a4a5 h5g7 a5a6 g7f5 e1f2 h7g8 g3g4 f5g7 b2b7 c7f7 c4a5 a8c8 a1a3 h6h5 a3b3 f6f5 b7f7 g6f7 b3b7 c8c7 a5c4 h5g4 h3g4 g7e8 c4e5 e8f6 f2e1 f7e8 e1a5 c7c8 g4f5 c8a8 f5e6 d5f4 g1f2 f6d5 a5d2 f4e6 f1c4 e6d8 b7b2 d8f7 c4d5 c6d5 e5f7 g8f7 d2g5 e8c6 g5f4 a8d8 f4e5 d8e8 f2e3 f7e6 b2h2 c6b5 h2b2 b5a6 b2a2 a6b5 a2a7 e8e7 a7a5 e7b7 e3f2 e6d7 a5a3 b5c4 a3a8 c4d3 a8a3 d3c4 f3f4 c4b5 f2g3 d7e6 g3g4 b7f7 g4g5 f7f5 g5g4 f5f8 a3a7 f8f5 a7a5 b5e2 g4g3 f5f8 g3f2 e2d3 a5a3 d3b5 a3b3 b5d7 b3b6 e6f5 f2f3 d7e6 c5c6 f8a8 c6c7 a8a3 f3e2 a3a2 e2d3 a2a3 d3c2 a3a4 c2b3 a4c4 b6b8 f5e4 b8h8 e6d7 h8d8 d7h3 d8h8 h3d7 h8h7 d7a4 b3a3 a4b5 h7h8 b5d7 h8h7 d7e6 a3b3 e6g4 h7g7 g4e6 g7g6 e6d7 g6g3 d7e6 b3b2 e4f5 g3g7 f5e4 b2b3 e6f5 g7g5 f5d7 g5g1 d7e6 g1g2 e6h3 g2g1 h3e6 g1g3 e6f5 b3b2 f5d7 g3a3 c4b4 b2c3 b4c4 c3d2 d7e6 f4f5 e4f5 a3a6 e6d7 a6a8 f5e6 a8h8 e6f7 d2e3 c4c3 e3f4 f7e7 h8h7 e7e6 f4g5 c3c1 g5g4 c1f1 g4g3 f1c1 g3f2 c1c3 f2e2 d7c8 h7h8 c8d7 e2d2 c3c4 h8h7 d7c8 d2e3 c4c1 h7h8 c8d7 e3f4 c1f1 f4g3 f1c1 g3f3 e6f5 h8h7 c1c3 f3e2 f5e6 e2f2 d7c8 h7h8 c8d7 f2e2 c3c2 e2f3 c2c1 h8h7 c1c2 h7g7 c2c1 f3e3 c1c2 g7g1 e6f5 g1g7 d7e6 g7h7 f5g5 h7h8 c2c3 e3d2 c3c4 h8e8 g5f5 d2e3 c4c3 e3f2 e6d7 e8d8 d7e6 d8h8 f5e4 h8f8 3388782 >Stockfish(1): isready 3388784 <Stockfish(1): readyok 3388785 >Stockfish(1): go wtime 7374 btime 8455 winc 5000 binc 5000 3388787 <Stockfish(1): info depth 1 seldepth 1 multipv 1 score cp 0 nodes 16924 nps 5641333 tbhits 101 time 3 pv e6f5 3388787 <Stockfish(1): info depth 2 seldepth 2 multipv 1 score cp 0 nodes 30597 nps 10199000 tbhits 111 time 3 pv e4d3 f2g3 3388787 <Stockfish(1): info depth 3 seldepth 4 multipv 1 score cp 0 nodes 39161 nps 9790250 tbhits 113 time 4 pv e4d3 f2g3 d3e4 g3h4 3388787 <Stockfish(1): info depth 4 seldepth 5 multipv 1 score cp 0 nodes 55290 nps 13822500 tbhits 127 time 4 pv c3c2 f2g3 e6f5 f8g8 c2c6 3388788 <Stockfish(1): info depth 5 seldepth 6 multipv 1 score cp 0 nodes 71757 nps 17939250 tbhits 156 time 4 pv c3c2 f2g3 e6f5 f8g8 c2c6 g3f2 3388788 <Stockfish(1): info depth 6 seldepth 8 multipv 1 score cp 0 nodes 91623 nps 22905750 tbhits 196 time 4 pv c3c2 f2e1 e4e3 e5f4 e3d3 f4d6 d3d4 e1d1 3388789 <Stockfish(1): info depth 7 seldepth 8 multipv 1 score cp 0 nodes 112455 nps 22491000 tbhits 230 time 5 pv c3c2 f2g3 e6f5 f8g8 c2c6 g3h4 f5d7 g8f8 3388789 <Stockfish(1): info depth 8 seldepth 10 multipv 1 score cp 0 nodes 138302 nps 27660400 tbhits 258 time 5 pv c3c2 f2g3 e6f5 f8g8 c2c3 g3h4 f5d7 g8d8 d7f5 3388790 <Stockfish(1): info depth 9 seldepth 15 multipv 1 score cp 0 nodes 169213 nps 28202166 tbhits 287 time 6 pv c3c2 f2g3 e6f5 f8g8 c2c3 g3h4 f5d7 g8a8 c3c2 a8f8 e4d3 f8f3 d3e4 3388790 <Stockfish(1): info depth 10 seldepth 13 multipv 1 score cp 0 nodes 199345 nps 33224166 tbhits 334 time 6 pv c3c2 f2g3 e6f5 f8a8 c2c6 a8d8 c6g6 g3f2 g6c6 3388792 <Stockfish(1): info depth 11 seldepth 17 multipv 1 score cp 0 nodes 273904 nps 34238000 tbhits 428 time 8 pv c3c2 f2g3 e6f5 f8a8 c2c6 a8d8 c6g6 g3f2 g6c6 3388792 <Stockfish(1): info depth 12 seldepth 12 multipv 1 score cp 0 nodes 324127 nps 36014111 tbhits 504 time 9 pv c3c2 f2e1 e4d3 f8d8 c2c1 e1f2 c1c6 c7c8q c6c8 d8c8 e6c8 3388793 <Stockfish(1): info depth 13 seldepth 18 multipv 1 score cp 0 nodes 371248 nps 37124800 tbhits 522 time 10 pv c3c2 f2g3 e6f5 g3h4 c2c6 f8h8 c6c1 h8e8 c1c6 3388795 <Stockfish(1): info depth 14 seldepth 21 multipv 1 score cp 0 nodes 476814 nps 43346727 tbhits 565 time 11 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 f3f4 e4d3 3388796 <Stockfish(1): info depth 15 seldepth 10 multipv 1 score cp 0 nodes 520519 nps 43376583 tbhits 583 time 12 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388797 <Stockfish(1): info depth 16 seldepth 10 multipv 1 score cp 0 nodes 572956 nps 44073538 tbhits 597 time 13 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388797 <Stockfish(1): info depth 17 seldepth 10 multipv 1 score cp 0 nodes 628682 nps 44905857 tbhits 673 time 14 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 f3f4 e4d3 3388798 <Stockfish(1): info depth 18 seldepth 10 multipv 1 score cp 0 nodes 657590 nps 46970714 tbhits 706 time 14 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 f3f4 e4d3 3388800 <Stockfish(1): info depth 19 seldepth 10 multipv 1 score cp 0 nodes 947852 nps 55756000 tbhits 909 time 17 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 f3f4 e4d3 3388801 <Stockfish(1): info depth 20 seldepth 10 multipv 1 score cp 0 nodes 996980 nps 58645882 tbhits 921 time 17 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388804 <Stockfish(1): info depth 21 seldepth 24 multipv 1 score cp 0 nodes 1415138 nps 70756900 tbhits 1237 time 20 pv e6d7 f8g8 d7e6 g8e8 e6d7 3388804 <Stockfish(1): info depth 22 seldepth 10 multipv 1 score cp 0 nodes 1473611 nps 70171952 tbhits 1334 time 21 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 f3f4 e4d3 3388805 <Stockfish(1): info depth 23 seldepth 10 multipv 1 score cp 0 nodes 1556219 nps 74105666 tbhits 1421 time 21 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388812 <Stockfish(1): info depth 24 seldepth 10 multipv 1 score cp 0 nodes 2338966 nps 83534500 tbhits 3936 time 28 pv e6d7 f8g8 d7e6 g8e8 e6d7 c7c8q c3c8 e8c8 d7c8 3388812 <Stockfish(1): info depth 25 seldepth 10 multipv 1 score cp 0 nodes 2435690 nps 83989310 tbhits 4193 time 29 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388813 <Stockfish(1): info depth 26 seldepth 10 multipv 1 score cp 0 nodes 2517502 nps 86810413 tbhits 4378 time 29 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388814 <Stockfish(1): info depth 27 seldepth 10 multipv 1 score cp 0 nodes 2602869 nps 86762300 tbhits 4567 time 30 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388814 <Stockfish(1): info depth 28 seldepth 10 multipv 1 score cp 0 nodes 2717081 nps 87647774 tbhits 4757 time 31 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 3388830 <Stockfish(1): info depth 29 seldepth 10 multipv 1 score cp 0 nodes 4583275 nps 99636413 tbhits 9680 time 46 pv e6d7 f2e2 c3c2 e2d1 e4d3 f8f3 d3e4 d1c2 e4f3 3388880 <Stockfish(1): info depth 30 seldepth 10 multipv 1 score cp 0 nodes 10430230 nps 108648229 tbhits 34263 time 96 pv e6d7 f8g8 d7e6 g8e8 e6d7 e8g8 3389373 <Stockfish(1): info depth 31 seldepth 34 multipv 1 score cp 0 nodes 69506721 nps 118008015 tbhits 325547 time 589 pv e6d7 f8g8 d7e6 g8h8 c3c2 f2g3 c2c3 g3f2 3389750 <Stockfish(1): info depth 32 seldepth 21 multipv 1 score cp 0 nodes 114014833 nps 118027777 tbhits 617498 time 966 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6d7 g3g7 d7a4 e5g3 c2c1 e1d2 c1c4 g7e7 e4f5 c7c8q c4c8 e7e5 f5f6 e5d5 3389943 <Stockfish(1): info depth 33 seldepth 31 multipv 1 score cp 0 nodes 136282480 nps 117586264 hashfull 34 tbhits 832750 time 1159 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 c2a2 g3g6 e4e3 e5f4 e3f4 g6e6 3390350 <Stockfish(1): info depth 34 seldepth 33 multipv 1 score cp 0 nodes 184127310 nps 117578103 hashfull 43 tbhits 1300590 time 1566 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6f5 e1d1 c2f2 g3g5 e4d3 g5f5 f2f5 3390411 <Stockfish(1): info depth 35 seldepth 19 multipv 1 score cp 0 nodes 191531751 nps 117720805 hashfull 44 tbhits 1362706 time 1627 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6f5 g3g8 e4d3 g8f8 f5d7 c7c8q d7c8 f8f3 d3e4 f3f2 c2f2 e1f2 3390772 <Stockfish(1): info depth 36 seldepth 16 multipv 1 score cp 0 nodes 234460468 nps 117937861 hashfull 49 tbhits 1713966 time 1988 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6f5 e5d6 c2a2 g3g5 a2a8 g5f5 e4f5 3390938 <Stockfish(1): info depth 37 seldepth 38 multipv 1 score cp 0 nodes 253269598 nps 117581057 hashfull 52 tbhits 1872304 time 2154 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6f5 g3a3 f5c8 a3a8 c8e6 c7c8q e6c8 e1d1 c2c3 d1d2 c3c6 a8b8 c8f5 b8f8 c6a6 d2c2 a6a2 c2c3 a2f2 f8d8 f5e6 d8e8 f2f3 c3b4 e6f5 e5h2 e4d4 3390979 <Stockfish(1): info depth 38 seldepth 32 multipv 1 score cp 0 nodes 257850863 nps 117471919 hashfull 53 tbhits 1910102 time 2195 pv e6d7 f8g8 d7e6 g8g3 c3c2 f2e1 e6f5 g3a3 f5c8 a3a8 c8e6 e1d1 e4d3 c7c8q c2c8 a8a3 d3e4 d1d2 e6d7 a3e3 e4f5 d2e1 f5g6 e3g3 g6f7 g3g7 f7e8 g7g8 e8e7 g8c8 d7c8 3390979 <Stockfish(1): info depth 73 seldepth 9 multipv 1 score cp 0 nodes 257859626 nps 117422416 hashfull 53 tbhits 1910167 time 2196 pv c3c2 f2e1 e6d7 c7c8q d7c8 f8f2 c2f2 e1f2 3390979 <Stockfish(1): bestmove c3c2 ponder f2e1

Alayan-stk-2 commented 4 years ago

Would this make any sense, conceptually (the actual code would look different) :

if (eval > 10 against self && depth > N) { // everything loses 
   don't select pawn push as best move;
   don't select captures as best move unless it's "free";
}

It doesn't gain elo in rating lists, it rarely would change anything when adjudication is used ; but on the odd occasion that SF sees it's lost and randomly pick a 50mr reset, it would make it so much less frustrating.

Arguably not worth the hassle, it's simply frustrating that when SF sees that everything loses, its defense becomes less challenging.

vondele commented 4 years ago

no wouldn't make much sense, e.g. the optimal move to give the longest path to mate could be a pawn push or a capture, or unless one captures next move is a fork on K and Q etc.... would be ugly very quickly. SF plays the best move, one can only try to improve its understanding of what is a bestmove.

adentong commented 4 years ago

@vondele Do you think there's anything else that can be done for this, other than the patch you just merged today? If not then I'll close the issue.

vondele commented 4 years ago

Let's keep it open for a few days, in case @joergoster sees any relationship with the commit mentioned previously. If not we can close. Thanks.

Alayan-stk-2 commented 4 years ago

SF plays the best move, one can only try to improve its understanding of what is a bestmove.

Yeah, I generally agree ; but in lost positions where everything is horrible, there is no good way to differentiate "challenging moves" from "delay mate longer but give easy play to the opponent". It doesn't seem really fixable.

joergoster commented 4 years ago

@vondele I thought it to be quite obvious now from my above posts, that one possible cause for this issue is the thread voting patch. See the highlighted PV line at the end of my 2nd post.

The question remains, though, why one thread (or even more than one?) keeps flying through the plies and reaching depth 73 without noticing that this root move is losing. It is possible that the patch you mentioned is causing this. OTOH, a single-threaded search doesn't show this problem. The short time-control may also play a role here, idk.

NKONSTANTAKIS commented 4 years ago

if (pos.rule50_count() < 90) return ttValue;

It seems to me very likely that 90 is more drastic than required and backfires, while the rare ill effects of GHI could be alleviated with smaller margin, say 94.

But the question is how to test? Normal TC's rarely trigger it. Maybe just on those 50-move positions?

vondele commented 4 years ago

@joergoster I indeed didn't see that you highlighted the interesting fact that the different threads must have had a difference of depth ~40 in that run. That's an indication of a potential issue, but one would need to understand better.

joergoster commented 4 years ago

@vondele Yes, it would certainly help to know whether this also happened in other cases.

Alayan-stk-2 commented 4 years ago

I must admit I didn't see this huge jump in depth at first as I skipped between the highlighted parts of the log. It's very suspicious, especially when considering seldepth...

The output shows a seldepth of 9 only. The line that was chosen in PV was not forced at all, so I fail to see the why of this abysmal seldepth. SF expected Ke1 and a quick exchange into a 7-men TB draw (this is a set of TB it must have at CCC).

The 50mr counter on Rc2+ was at 77, so there is a good chance that this bug isn't directly caused by 50mr.

adentong commented 4 years ago

Happened again in game 94 of SUFI. Leela was completely clueless and had no idea how to convert the endgame, but SF pushed a pawn which helped Leela. Even if the endgame was objectively lost, I strongly believe that if SF didn't push the pawn it woulda drawn the game due to Leela's cluelessness. We need a patch that ignores pawn moves in completely lost positions. This patch can be tested for non regression and applied for tournament play only. I imagine doing so definitely won't hurt since if a position is already completely lost then it doesn't matter what move is played, but it'll definitely exploit Leela's bad endgames and possibly draw some otherwise lost endgames. @vondele @Alayan-stk-2

vondele commented 4 years ago

@adentong link, fen, move played, better move, + deep analysis to correctly assess the position. In this case, the cutechess log would be useful as well (to check the actual depth etc).

nickolasreynolds commented 4 years ago

TCEC S17 Sufi Game 94, position after 143. Ra1: 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143.

TCECfish played the instantly suicidal Nd7, supposedly after evaluating 530 million nodes. The better move was Kd7. My 20200407 Homefish quickly switches away from Nc7 / c2 / Nd4+ (moves it does initially consider at low depths), after which it prefers Kd7 forever. The linked Lifish also mirrors my Homefish's behavior.

adentong commented 4 years ago

Even after move 163, when the position is already objectively lost 5k2/1R6/4p1n1/4PpP1/3K4/8/8/8 b - - 10 163, pushing f4 is essentially just conceding. Even if in this particular case it may not have mattered at that point not pushing the pawn would have been stauncher defense. Also @nickolasreynolds seems like in the other issue I opened https://github.com/official-stockfish/Stockfish/issues/2643 SF also chose an instantly losing move which homefish very quickly refutes but SF with 176 threads somehow fail to do so.

GeorgeMJ23 commented 4 years ago

I also tried Stockfish 070420 x64 at 64 threads 5 times and never considered 143...Nc7? blunder that throws the game. It just plays Kd7 from start to finish. After 143...Nc7 it shows fail low after 2 seconds with >+2.50 evals and after 10 seconds goes to +4.5. I use partial 7 men TBs though and don't know if it makes a difference.

143...Kd7 appears to be a sure draw.

Analysis by Stockfish 070420 64 POPCNT: 143...Ke7-d7 144.Ra1-h1 Kd7-e8 145.Kb3-b4 Nb5-c7 146.Kb4xc3 Nc7xa6 147.Rh1-h8+ Ke8-e7 148.Rh8-h7 Ke7-e8 149.g5-g6 f7xg6 150.Rh7xa7 Na6-c5 151.Ra7-g7 g6-g5 152.f4xg5 Ke8-f8 153.Rg7-c7 Nc5-e4+ 154.Kc3-d4 Ne4xg5 155.Kd4-e3 Kf8-e8 156.Rc7-c6 Ke8-e7 157.Rc6xb6 Ke7-d7 158.Ke3-f4 Ng5-e4 159.Rb6-a6 Ne4-c5 160.Ra6-d6+ Kd7-e7 161.Rd6-d4 Nc5-e4 162.Rd4-a4 Ke7-d8 163.Kf4-f3 Kd8-d7 164.Ra4-a6 Ne4-g5+ 165.Kf3-f4 Ng5-e4 166.Ra6-b6 Ne4-f2 167.Rb6-d6+ Kd7-e7 168.Kf4-f3 Nf2-e4 169.Rd6-b6 Ke7-d7 170.Rb6-b8 Ne4-g5+ 171.Kf3-f4 Ng5-e4 172.Rb8-h8 Kd7-e7 173.Rh8-h3 Ke7-e8 174.Rh3-f3 Ke8-f7 ................ White is better: +/- (0.73) Depth: 76/87 00:02:54 10212MN, tb=105077340

The TCEC log is this: SF seems to think between 143...c2 and 143...Nc7 that were both losing moves.

786578331 >Stockfish 20200407DC(137): isready 786579048 <Stockfish 20200407DC(137): readyok 786579048 >Stockfish 20200407DC(137): go wtime 25230 btime 32864 winc 5000 binc 5000 786579065 <Stockfish 20200407DC(137): info depth 1 seldepth 1 multipv 1 score cp -174 nodes 52987 nps 7569571 tbhits 0 time 7 pv c3c2 786579065 <Stockfish 20200407DC(137): info depth 2 seldepth 3 multipv 1 score cp -174 nodes 70171 nps 10024428 tbhits 0 time 7 pv b5c7 b3c3 786579066 <Stockfish 20200407DC(137): info depth 3 seldepth 3 multipv 1 score cp -174 nodes 83619 nps 11945571 tbhits 0 time 7 pv b5c7 b3c3 e7d7 786579066 <Stockfish 20200407DC(137): info depth 4 seldepth 5 multipv 1 score cp -174 nodes 109478 nps 15639714 tbhits 0 time 7 pv e7d7 a1h1 d7e7 786579066 <Stockfish 20200407DC(137): info depth 5 seldepth 5 multipv 1 score cp -174 nodes 126250 nps 15781250 tbhits 0 time 8 pv e7d7 a1h1 d7e7 h1h5 e7d7 786579067 <Stockfish 20200407DC(137): info depth 6 seldepth 7 multipv 1 score cp -167 nodes 145695 nps 18211875 tbhits 0 time 8 pv b5c7 b3c3 c7d5 c3c4 d5c7 a1h1 c7a6 786579068 <Stockfish 20200407DC(137): info depth 7 seldepth 10 multipv 1 score cp -163 nodes 181837 nps 22729625 tbhits 0 time 8 pv b5c7 b3c3 e7d7 c3c4 d7e8 a1h1 c7a6 h1h8 e8e7 h8a8 786579069 <Stockfish 20200407DC(137): info depth 8 seldepth 9 multipv 1 score cp -174 nodes 224347 nps 28043375 tbhits 0 time 8 pv b5c7 b3c3 e7d7 c3c4 d7e8 a1a3 e8e7 a3d3 c7a6 786579070 <Stockfish 20200407DC(137): info depth 9 seldepth 16 multipv 1 score cp -174 nodes 272461 nps 30273444 tbhits 0 time 9 pv b5c7 b3c3 e7d7 c3b3 d7e7 b3c4 e7e8 a1a3 e8e7 a3d3 e7e8 d3h3 c7a6 h3h8 e8e7 h8a8 786579071 <Stockfish 20200407DC(137): info depth 10 seldepth 11 multipv 1 score cp -174 nodes 309930 nps 34436666 tbhits 0 time 9 pv b5c7 b3c3 e7d7 c3c4 d7e8 a1a3 e8e7 a3d3 c7a6 d3a3 a6c7 786579072 <Stockfish 20200407DC(137): info depth 11 seldepth 15 multipv 1 score cp -174 nodes 354694 nps 35469400 tbhits 0 time 10 pv b5c7 b3c3 e7d7 c3c4 d7e7 a1c1 e7e8 c1h1 c7a6 h1h8 e8e7 h8a8 e7d7 a8a7 a6c7 786579073 <Stockfish 20200407DC(137): info depth 12 seldepth 21 multipv 1 score cp -174 nodes 430998 nps 43099800 tbhits 0 time 10 pv b5c7 b3c3 e7d7 c3c4 d7e7 a1h1 c7a6 h1a1 a6c7 a1a7 e7d7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 b5b4 f8f7 b4b3 786579075 <Stockfish 20200407DC(137): info depth 13 seldepth 22 multipv 1 score cp -174 nodes 533502 nps 48500181 tbhits 0 time 11 pv b5c7 b3c3 e7d7 c3c4 d7e7 a1a3 e7e8 a3h3 c7a6 h3h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 786579077 <Stockfish 20200407DC(137): info depth 14 seldepth 18 multipv 1 score cp -184 nodes 629173 nps 52431083 tbhits 0 time 12 pv b5c7 b3c3 e7f8 c3b3 f8e7 b3c4 e7e8 a1a3 e8e7 a3h3 c7a6 h3a3 a6c7 a3a7 e7d7 a7b7 d7c6 786579079 <Stockfish 20200407DC(137): info depth 15 seldepth 25 multipv 1 score cp -201 nodes 1248502 nps 69361222 tbhits 0 time 18 pv b5c7 b3c3 e7f8 c3b3 f8e7 b3c4 e7e8 a1a3 e8f8 a3a2 f8e7 a2d2 e7e8 d2h2 c7a6 h2h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7b7 d7c6 b7b8 c7d5 786579080 <Stockfish 20200407DC(137): info depth 16 seldepth 17 multipv 1 score cp -201 nodes 1319114 nps 73284111 tbhits 0 time 18 pv b5c7 b3c3 e7f8 c3b3 f8e7 b3c4 e7e8 a1a2 e8f8 a2d2 f8e8 d2h2 c7a6 h2h8 e8e7 h8a8 e7d7 786579122 <Stockfish 20200407DC(137): info depth 17 seldepth 24 multipv 1 score cp -210 nodes 9509728 nps 125128000 tbhits 7 time 76 pv e7d7 a1h1 d7e7 h1h4 e7f8 h4h8 f8e7 b3c2 e7d7 h8h2 d7e7 h2h3 e7f8 h3h8 f8e7 c2d1 e7d7 h8a8 d7e7 a8c8 e7d7 786579143 <Stockfish 20200407DC(137): info depth 18 seldepth 28 multipv 1 score cp -230 nodes 12915539 nps 134536864 tbhits 8 time 96 pv e7d7 a1h1 d7e8 h1h2 e8e7 h2h7 e7f8 h7h8 f8e7 b3c2 e7d7 h8h3 d7e7 h3h1 e7e8 h1h2 e8f8 h2h8 f8e7 h8a8 e7d7 786579166 <Stockfish 20200407DC(137): info depth 19 seldepth 24 multipv 1 score cp -197 nodes 17192060 nps 143267166 tbhits 8 time 120 pv e7d8 a1h1 d8e8 h1h7 e8f8 h7h8 f8e7 b3c2 e7d7 h8h3 d7e7 h3h2 e7f8 h2h8 f8e7 h8c8 e7d7 786579179 <Stockfish 20200407DC(137): info depth 20 seldepth 27 multipv 1 score cp -185 nodes 19047160 nps 147652403 tbhits 8 time 129 pv e7d8 a1d1 d8e8 d1h1 e8e7 h1h5 e7d8 h5h2 d8e8 h2h7 e8f8 h7h8 f8e7 b3c2 e7d7 h8a8 d7e7 a8c8 e7d7 c8h8 d7e7 h8h4 e7d7 786579214 <Stockfish 20200407DC(137): info depth 21 seldepth 39 multipv 1 score cp -202 nodes 26476569 nps 158542329 tbhits 9 time 167 pv b5c7 b3c3 c7d5 c3d4 d5c7 d4d3 e7e8 a1a2 e8d7 d3c4 d7e7 a2h2 c7a6 h2a2 a6c7 a2a7 e7d7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 b5b4 f8f7 b4b3 f7h7 d5f4 786579222 <Stockfish 20200407DC(137): info depth 22 seldepth 29 multipv 1 score cp -174 nodes 28138113 nps 160789217 tbhits 9 time 175 pv b5c7 b3c3 e7d7 c3c4 d7e8 a1a3 e8e7 c4b3 e7e8 a3a2 e8e7 b3b4 e7d7 b4c4 d7e7 a2h2 c7a6 h2a2 a6c7 a2a7 e7d7 a7a3 d7c6 a3h3 786579265 <Stockfish 20200407DC(137): info depth 23 seldepth 37 multipv 1 score cp -174 nodes 37292309 nps 170284515 tbhits 181 time 219 pv b5c7 b3c3 e7e8 c3c4 e8f8 a1d1 f8e8 d1h1 c7a6 h1h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7b7 d7c6 b7b8 c7d5 b8f8 b6b5 c4d4 b5b4 f8f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 f4g6 b3b7 e7f8 786579317 <Stockfish 20200407DC(137): info depth 24 seldepth 51 multipv 1 score cp -209 nodes 48943422 nps 180603033 tbhits 343 time 271 pv b5c7 b3c3 e7d7 c3d4 d7e7 a1a2 e7f8 d4c3 f8e8 c3c4 e8e7 a2h2 c7a6 h2a2 a6c7 a2a7 e7d7 a7a1 d7c6 a1h1 c7d5 h1h7 b6b5 c4d4 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e8 b1b3 786579346 <Stockfish 20200407DC(137): info depth 25 seldepth 39 multipv 1 score cp -177 nodes 55386393 nps 184621310 tbhits 588 time 300 pv b5c7 b3c3 e7f8 c3b4 f8e7 b4c4 e7e8 c4d3 e8e7 a1a2 e7d7 d3d4 d7e7 d4c3 c7b5 c3c4 b5c7 a2h2 c7a6 h2a2 a6c7 a2a7 e7d7 786579381 <Stockfish 20200407DC(137): info depth 26 seldepth 52 multipv 1 score cp -174 nodes 62913770 nps 187802298 tbhits 670 time 335 pv b5c7 b3c3 e7e8 a1a2 e8d7 c3b3 d7e7 b3c4 e7f8 a2d2 f8e8 d2d6 e8e7 d6d3 e7e8 d3h3 c7a6 h3h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 b5b4 f8f7 786579407 <Stockfish 20200407DC(137): info depth 27 seldepth 47 multipv 1 score cp -173 nodes 68709640 nps 190331412 tbhits 889 time 361 pv b5c7 b3c3 c7d5 c3c4 d5c7 a1d1 c7a6 d1a1 a6c7 a1a7 e7d7 a7a1 d7c6 a1h1 b6b5 c4d3 c7d5 h1h7 d5f4 d3d4 b5b4 h7f7 b4b3 f7f8 c6d7 f8b8 d7e7 b8b3 e7f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7f8 h7a7 f8g8 a7a3 g8f7 a3f3 f7g7 f3h3 786579481 <Stockfish 20200407DC(137): info depth 28 seldepth 48 multipv 1 score cp -171 nodes 85082908 nps 196043566 tbhits 2533 time 434 pv b5c7 b3c3 c7d5 c3c4 d5c7 a1h1 c7a6 h1a1 a6c7 a1a7 e7d7 a7a3 d7c6 a3h3 c7d5 h3h7 b6b5 c4d3 b5b4 h7f7 d5f4 d3d4 b4b3 f7f8 c6d7 f8b8 d7e7 b8b7 e7f8 b7b3 f8g7 b3f3 f4g6 f3f1 g7f7 f1h1 f7g7 786579503 <Stockfish 20200407DC(137): info depth 29 seldepth 54 multipv 1 score cp -165 nodes 90289404 nps 197569811 tbhits 3367 time 457 pv b5c7 b3c3 e7d7 c3b4 d7e8 a1a3 e8e7 b4c4 e7f8 a3d3 f8e7 d3h3 c7a6 h3a3 a6c7 a3a7 e7d7 a7a3 d7c6 a3h3 b6b5 c4d4 b5b4 h3h7 c7d5 h7f7 b4b3 f7h7 d5f4 h7h8 c6d7 h8b8 d7e7 b8b3 e7f7 b3f3 f4g6 f3h3 f7f8 h3h7 f8g8 h7c7 g6h4 c7c3 h4g6 c3b3 g8g7 b3b7 g7g8 b7b1 g8f7 b1c1 g6f4 786579546 <Stockfish 20200407DC(137): info depth 30 seldepth 55 multipv 1 score cp -151 nodes 100092377 nps 200585925 tbhits 4985 time 499 pv c3c2 b3c2 b5c7 c2b3 e7d7 a1a2 d7e8 b3c3 e8d7 c3d4 d7e7 d4c4 e7f8 a2a3 f8e7 a3d3 c7a6 d3a3 a6c7 a3a7 e7d7 a7a2 d7c6 a2h2 c7d5 h2h7 b6b5 c4d4 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 e7f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7g8 h7a7 g6h4 a7a8 g8g7 786579671 <Stockfish 20200407DC(137): info depth 31 seldepth 57 multipv 1 score cp -150 nodes 128980599 nps 206699677 tbhits 11474 time 624 pv c3c2 b3c2 b5c7 c2b3 e7f8 b3c3 c7d5 c3c4 d5c7 a1a2 f8e7 a2h2 c7a6 h2a2 a6c7 a2a7 e7d7 a7a3 d7c6 a3h3 b6b5 c4d4 b5b4 h3h7 c7d5 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 e7f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7g8 h7b7 g6h4 d4e3 h4g6 b7b8 g8g7 b8b6 g7f7 e3d4 786579685 <Stockfish 20200407DC(137): info depth 32 seldepth 58 multipv 1 score cp -150 nodes 132817621 nps 207852302 tbhits 11635 time 639 pv c3c2 b3c2 b5c7 c2b3 e7d7 a1a2 d7e8 a2a3 e8d7 b3c4 d7e8 a3h3 c7a6 h3h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7a3 d7c6 a3h3 c7d5 h3h7 b6b5 c4d4 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 e7f7 b3f3 f4g6 f3h3 f7f8 h3h7 f8g8 h7b7 g8f8 b7c7 f8g8 c7c3 g8g7 c3d3 g7f7 d3f3 f7e7 f3c3 e7f7 c3c6 g6f4 786579731 <Stockfish 20200407DC(137): info depth 33 seldepth 60 multipv 1 score cp -150 nodes 144757566 nps 211324913 tbhits 15068 time 685 pv c3c2 b3c2 b5c7 c2b2 e7d7 b2b3 d7e8 b3c4 e8e7 a1a2 e7f8 a2d2 f8e7 d2h2 c7a6 h2a2 a6c7 a2a7 e7d7 a7a1 d7c6 a1h1 b6b5 c4d4 c7d5 h1h7 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e8 b1b3 f4g6 b3h3 e8f8 h3a3 f8g8 a3a1 g8f7 a1a7 f7g8 a7c7 g6h4 d4e3 h4g6 c7c5 g8f7 c5b5 f7g7 b5b6 g7f7 e3d4 786579928 <Stockfish 20200407DC(137): info depth 34 seldepth 59 multipv 1 score cp -149 nodes 196024524 nps 222502297 tbhits 30711 time 881 pv c3c2 b3c2 b5c7 c2d3 e7e8 d3c3 c7d5 c3c4 d5c7 a1a2 e8e7 a2c2 e7e8 c2h2 c7a6 h2h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7a2 d7c6 a2h2 b6b5 c4d4 b5b4 h2h7 c7d5 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e8 b1b3 f4g6 b3h3 e8f7 h3h7 f7g8 h7a7 g6h4 a7a3 h4g6 a3b3 g8f7 786580010 <Stockfish 20200407DC(137): info depth 35 seldepth 60 multipv 1 score cp -139 nodes 218126912 nps 226507696 tbhits 33512 time 963 pv c3c2 b3c2 b5c7 c2d3 e7e8 a1a3 c7d5 d3c4 d5c7 a3h3 c7a6 h3h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7a2 d7c6 a2h2 b6b5 c4d4 b5b4 h2h7 c7d5 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 e7f7 b3f3 f4g6 f3h3 f7f8 h3h7 f8g8 h7a7 g6h4 a7a8 g8g7 d4e3 h4g2 e3f3 g2h4 f3g3 h4g6 a8a5 g7g8 a5b5 g8f8 g3f3 f8e8 b5a5 e8f7 a5c5 786580103 <Stockfish 20200407DC(137): info depth 36 seldepth 64 multipv 1 score cp -139 nodes 242983113 nps 230097644 hashfull 20 tbhits 43191 time 1056 pv c3c2 b3c2 b5c7 c2d3 e7e8 a1a3 c7d5 d3c4 d5c7 a3h3 c7a6 h3h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7a2 d7c6 a2h2 b6b5 c4d4 c7d5 h2h7 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 f4g6 h1b1 c6d7 b1b3 d7e7 b3b7 e7f8 b7a7 f8g8 a7a5 g8f7 d4e3 f7g8 a5c5 g8f7 c5c7 f7f8 c7c8 f8g7 c8c6 g7f7 e3d4 g6f4 c6d6 f7e7 786580129 <Stockfish 20200407DC(137): info depth 37 seldepth 57 multipv 1 score cp -139 nodes 249923686 nps 230983073 hashfull 20 tbhits 46992 time 1082 pv c3c2 b3c2 b5c7 c2d3 c7d5 a1h1 d5b4 d3c4 b4a6 h1a1 a6c7 a1a7 e7d7 a7a2 d7c6 a2h2 b6b5 c4d4 c7d5 h2h7 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 f4g6 h1b1 c6d7 b1b3 d7e7 b3b7 e7f8 b7a7 f8g8 a7a5 g6f4 d4e3 f4g6 a5c5 g8g7 c5b5 g7f7 b5b7 f7e8 b7b6 e8f7 e3d4 g6f4 b6d6 f7e7 d6a6 e7f7 d4e3 f4g6 a6a7 f7f8 786580699 <Stockfish 20200407DC(137): info depth 38 seldepth 64 multipv 1 score cp -141 nodes 397032609 nps 240189116 hashfull 24 tbhits 218151 time 1653 pv c3c2 b3c2 b5c7 c2d3 c7d5 d3d4 d5c7 a1a2 e7f8 d4c4 f8e7 a2d2 c7a6 d2a2 a6c7 a2a7 e7d7 a7a1 d7c6 a1h1 c7d5 h1h7 b6b5 c4d4 b5b4 h7f7 b4b3 f7h7 d5f4 h7h8 c6d7 h8b8 d7e7 b8b3 e7f7 b3f3 f4g6 f3f2 f7g8 f2h2 g8g7 h2h6 g7f7 h6h7 f7g8 h7c7 g6h4 c7c8 g8g7 d4e3 h4g6 c8c5 g7g8 c5b5 g8f7 b5b7 f7g8 b7b6 g8f7 786580792 <Stockfish 20200407DC(137): info depth 39 seldepth 68 multipv 1 score cp -139 nodes 421775755 nps 241566869 hashfull 24 tbhits 228296 time 1746 pv c3c2 b3c2 b5c7 c2d2 e7d8 d2d3 c7d5 a1f1 d8e8 f1h1 d5b4 d3c4 b4a6 h1h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7a2 d7c6 a2h2 b6b5 c4d4 c7d5 h2h7 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e8 b1b3 e8f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7g8 h7c7 g6h4 c7c3 h4g6 c3e3 g8f8 d4c5 f8e7 e3e2 e7d7 786580832 <Stockfish 20200407DC(137): info depth 40 seldepth 65 multipv 1 score cp -139 nodes 432425321 nps 242255081 hashfull 24 tbhits 233193 time 1785 pv c3c2 b3c2 b5c7 c2d2 e7d8 d2d3 d8e8 a1a2 e8e7 d3d4 e7f8 d4c4 f8e7 a2a3 e7f8 c4b4 f8e8 b4b3 e8d7 b3c4 d7e8 a3h3 c7a6 h3h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 b5b4 f8f7 b4b3 f7h7 d5f4 h7h2 c6d7 h2b2 d7e7 b2b3 e7f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7g8 h7b7 g8f8 b7a7 f8g8 a7a8 g8f7 a8a6 g6f4 d4e3 f4g6 786581036 <Stockfish 20200407DC(137): info depth 41 seldepth 63 multipv 1 score cp -130 nodes 486555523 nps 244623189 hashfull 26 tbhits 268811 time 1989 pv c3c2 b3c2 b5c7 c2d3 e7e8 a1a3 c7d5 d3c4 d5c7 a3h3 c7a6 h3h8 e8e7 h8a8 e7d7 a8a7 a6c7 a7a1 d7c6 a1h1 c7d5 h1h7 b6b5 c4d4 b5b4 h7f7 b4b3 f7h7 d5f4 h7h1 c6d7 h1b1 d7e7 b1b3 e7f7 b3f3 f4g6 f3h3 f7g7 h3h6 g7f7 h6h7 f7g8 h7a7 g6h4 a7a8 g8g7 d4e3 h4g6 a8a5 g7f7 a5c5 f7g8 e3f3 g8f7 c5b5 g6h4 f3f4 h4g6 f4g3 f7f8 b5b7 f8g8 786581197 <Stockfish 20200407DC(137): info depth 42 seldepth 61 multipv 1 score cp -130 nodes 529425683 nps 246244503 hashfull 27 tbhits 300301 time 2150 pv c3c2 b3c2 b5c7 c2d2 e7d7 d2d3 c7d5 a1f1 d5b4 d3c4 b4a6 f1h1 d7e8 h1h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7a1 d7c6 a1h1 b6b5 c4d4 c7d5 h1h7 b5b4 h7f7 b4b3 f7h7 d5f4 h7h2 c6d7 h2b2 d7e7 b2b3 e7f7 b3f3 f4g6 f3h3 f7f8 h3h7 f8g8 h7c7 g6h4 c7c3 h4g6 c3e3 g8f7 d4c5 f7e7 e3e1 e7d7 c5d4 d7e7 e1a1 e7f7 a1a2 g6f4 d4e3 f4g6 786581200 <Stockfish 20200407DC(137): info depth 42 seldepth 63 multipv 1 score cp -130 nodes 530002098 nps 246169111 hashfull 27 tbhits 300892 time 2153 pv b5c7 b3c3 c7b5 c3b4 b5c7 a1c1 e7d7 c1h1 c7a6 b4c4 d7e8 h1h8 e8e7 h8a8 a6c7 a8a7 e7d7 a7b7 d7c6 b7b8 b6b5 c4d4 c7d5 b8f8 b5b4 f8f7 b4b3 f7h7 d5f4 h7h2 c6d7 h2b2 d7e7 b2b3 e7f7 b3f3 f4g6 f3h3 f7f8 h3h7 f8g8 h7c7 g6h4 c7c3 h4g6 c3e3 g8f7 d4c5 f7e7 e3e1 e7d7 c5d4 d7e7 e1a1 e7f7 a1a4 g6f4 d4c5 f4h3 g5g6 f7g6 c5d6 h3g5 786581201 <Stockfish 20200407DC(137): bestmove b5c7 ponder b3c3

adentong commented 4 years ago

@vondele So there have now been at least two instances where SF chose a losing move which homefish, supposedly on inferior hardware, refutes quickly. Could it be that 176 threads are too many threads and SF doesn't scale well past a certain number of threads?

adentong commented 4 years ago

Also https://github.com/official-stockfish/Stockfish/issues/2620#issuecomment-612852816 mentioned above. I think we should take a look at it to see if it's a legitimate issue.

vondele commented 4 years ago

@adentong I'm actually currently running tests on the fen posted by @nickolasreynolds (thanks!). I'll report a bit later. The question is not if sf can refute the move (of course it can), but if it always finds the move. So, can your run the search (e.g. for 1sec) on homefish ~200 times, and see what the distribution of bestmoves is. I'm doing that right now on 250 threads.

adentong commented 4 years ago

@vondele Will do. I'll report back once I finish.

joergoster commented 4 years ago

@vondele From the log snippet provided it looks like a thread has been picked which didn't find the refutation of Nc5-b7 in the given time.

vondele commented 4 years ago

@joergoster yes, question is why... I can reproduce similar results locally, running 10s of repetitions with following settings (changing movetime and # threads), up to 5 different moves are picked (4 of which are wrong):

setoption name Threads value 250
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 1000

# count #bestmove    #analysis (200s)
    216      e7d7 ->    cp 110
     44      e7d8 ->    cp 471
      7      e7e8 ->    cp 421
      5      b5c7 ->    cp 400

setoption name Threads value 250
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 2153

# count #bestmove
    141      e7d7
     65      e7d8
      2      b5c7

setoption name Threads value 50
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 2153

# count #bestmove
    197      e7d7
     11      b5c7

setoption name Threads value 50
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 10000

# count #bestmove
     65 bestmove e7d7
      2 bestmove b5c7
      1 bestmove c3c2  -> cp 400

the situation seems worse with many threads, but that's maybe coincidence. I wonder if there are just many threads with the wrong move as a search result, or if there is a different reason. Since it seems to be reproducible, maybe it can be analyzed.

Edit: the distribution depends a bit on the version, the version I used was about 2weeks old, current master gives:

setoption name Threads value 250
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 1000

    160 bestmove e7d7 
     13 bestmove b5c7
     12 bestmove e7d8 
      5 bestmove e7e8
NKONSTANTAKIS commented 4 years ago

This is imo the danger of too many reductions on multicore. They definitely help at 20+0.2" 8-thread, but its very risky for high threads + gains should diminish at high TC, because depth is much more needed at low TC's. In general I think there is an issue with multithreaded LTC reliability.

Probably though this is something more buggy than just reductions.

joergoster commented 4 years ago

@vondele Thank you, very interesting! Although I think the main issue here is the very small amount of time available, probably all 178 threads want to probe the hash table and the TBs almost simultaneously, can you please also check with my 50-move rule patch https://github.com/official-stockfish/Stockfish/commit/bae019b53e5c2bfcf0d69b4ebfc52b4f4de762eb removed?

NKONSTANTAKIS commented 4 years ago

My apologies

vondele commented 4 years ago

@NKONSTANTAKIS yes your need to relax about this. If you want to comment things like that, please, take the code, make the changes, run the tests. Otherwise it isn't particularly helpful.

vondele commented 4 years ago

@joergoster with the revert of that patch on master results look similar (250th, 1000ms search):

      4 bestmove b5c7 
     87 bestmove e7d7 
      4 bestmove e7d8 
      3 bestmove e7e8
joergoster commented 4 years ago

@vondele Thanks, much appreciated! I just wanted to be sure this patch of mine is not the root cause. ;-)

NKONSTANTAKIS commented 4 years ago

@vondele I just hope what I'm doing its not seriously obstructing. If my posts are 90% useless and 10% of some use, I am happy. But I have to be careful to not put myself on auto-ignore, else this 10% will pass unnoticed...

All in all, I am living it ;)

vondele commented 4 years ago

out_ana.txt

uploaded a file with 200 searches on this fen with 250 threads. The code has a modification so that one can see the bestmove for each thread at voting time together with depth, and score. 

diff --git a/src/search.cpp b/src/search.cpp
index a7e90a0..31f7b69 100644
--- a/src/search.cpp
+++ b/src/search.cpp
@@ -284,6 +284,10 @@ void MainThread::search() {
       for (Thread* th: Threads)
           minScore = std::min(minScore, th->rootMoves[0].score);

+      for (Thread* th: Threads) {
+          std::cout << "xxx " << UCI::move(th->rootMoves[0].pv[0], rootPos.is_chess960()) << " " << th->completedDepth << " " << th->rootMoves[0].sc
+      }
+
       // Vote according to score and depth, and select the best thread
       for (Thread* th : Threads)
       {

for example for one search, sorted with equal entries counted:

      4 xxx b5c7 36 -441 504
      1 xxx b5d4 36 -441 504
      1 xxx c3c2 33 -406 1617
      2 xxx c3c2 34 -406 1666
      2 xxx c3c2 35 -406 1715
      1 xxx b5c7 35 -403 1820
      1 xxx c3c2 37 -370 3145
      1 xxx b5c7 32 -355 3200
      1 xxx b5c7 36 -366 3204
      1 xxx b5d4 33 -357 3234
      1 xxx b5c7 37 -367 3256
      1 xxx b5c7 37 -366 3293
      1 xxx b5d4 37 -366 3293
      1 xxx c3c2 37 -366 3293
      1 xxx c3c2 34 -349 3604
      1 xxx b5d4 35 -349 3710
      3 xxx b5c7 35 -348 3745
      2 xxx b5d4 35 -348 3745
      5 xxx b5d4 36 -349 3816
      5 xxx c3c2 36 -349 3816
      1 xxx b5d4 36 -348 3852
      4 xxx c3c2 36 -348 3852
     12 xxx b5c7 37 -349 3922
      8 xxx b5d4 37 -349 3922
      9 xxx c3c2 37 -349 3922
      1 xxx b5c7 36 -346 3924
     19 xxx b5c7 37 -348 3959
      2 xxx b5d4 37 -348 3959
      9 xxx c3c2 37 -348 3959
     30 xxx b5c7 38 -349 4028
      2 xxx b5d4 38 -349 4028
     13 xxx c3c2 38 -349 4028
     44 xxx b5c7 38 -348 4066
      3 xxx c3c2 38 -348 4066
     19 xxx b5c7 39 -349 4134
     38 xxx b5c7 39 -348 4173

no single thread stands out, they're just all wrong.

NKONSTANTAKIS commented 4 years ago

Lets try with less logarithmic reductions?

nickolasreynolds commented 4 years ago

Is it possible / reasonable to gradually reduce reductions and increase pruning thresholds for each new thread as the number of threads increases?

silversolver1 commented 4 years ago

Not trying to point any fingers here, since it is still not very clear to me which compile was sent of which version of SF, but since it looks like it was a compile after 4/2/20, maybe #2603 could have played a role in this? Master still currently has the following line: https://github.com/official-stockfish/Stockfish/blob/221893bf679f70098e6f751fded2fe843471c6be/src/search.cpp#L300 Are we sure that this is not supposed to be a less than or equal to sign instead of a greater than sign?

It would also help if we could see the diff of the version that was actually sent

EDIT: replaced incorrect link with actual link to line in current master EDIT 2: could also potentially be related to #2643

vondele commented 4 years ago

just for reference, so that we don't point to random patches in the past, the results for SF 10 are clearly worse than master, so it is definitely not some recent regression:

setoption name Threads value 250
setoption name Hash value 80000
position fen 8/p3kp2/Pp2p3/1n2PpP1/5P2/1Kp5/8/R7 b - - 68 143
go movetime 1000

     32 bestmove b5c7 
      6 bestmove b5d4 
     73 bestmove e7d7 
     97 bestmove e7d8

and similar for SF 9:

     34 bestmove b5c7 
     16 bestmove c3c2 
     96 bestmove e7d7 
     60 bestmove e7d8 
      1 bestmove e7e8 
      1 bestmove e7f8
nickolasreynolds commented 4 years ago

Is this another example?

TCEC S17 Sufi Game 95, position after 32. ... Be7: 5r2/1k2b2p/2q1p3/Pp1bPrpB/2pP4/6QP/2RB1PP1/5RK1 w - - 2 33.

TCECfish played the instantly suicidal Rb2, supposedly after evaluating 28 billion (!) nodes. My 20200407 Homefish quickly (0.0s) switches away from Rb2 , after which it prefers Be2 and then Bg4 forever. I only ran one test, perhaps people with beefier machines can do multiple runs up to 28 billion nodes to confirm that my Homefish's behavior is typical of non-supercomputers.

(It's also notable that Stockfish lost both sides of this opening, something that really shouldn't be possible at this level and time control.)

nickolasreynolds commented 4 years ago

And another possible example from the forums:

bsda...:

Looks like SF played another weird suicide move in TCEC game 98 https://www.tcec-chess.com/archive.html?season=17&div=sf&game=98

On move 30 SF plays a5 ?? then shows a losing eval 2 moves later

FEN: r1rb2k1/1bpnqn2/pp1p4/3Pp1p1/PP2PpPp/NQNB1P1P/5B2/R1R3K1 b - - 10 30

Bryan:

The latest SFdev only wants a5 at d=2, and then prefers other moves to d=50 and 39 billion moves. So I can't replicate a5 in the latest SFdev.

GeorgeMJ23 commented 4 years ago

TCECfish played the instantly suicidal Rb2, supposedly after evaluating 28 billion (!) nodes. My 20200407 Homefish quickly (0.0s) switches away from Rb2 , after which it prefers Be2 and then Bg4 forever. I only ran one test, perhaps people with beefier machines can do multiple runs up to 28 billion nodes to confirm that my Homefish's behavior is typical of non-supercomputers.

Things are not that easy, as running SF with e.g 50 threads it really likes Rb2 after a lot of time and 14 billion nodes(too lazy to wait another 4.5 minutes) believing it's not suicidal at all:

Analysis by Stockfish 070420 64 POPCNT:

33.Rc2-b2 Kb7-a8 34.Kg1-h2 Rf8-b8 35.Bd2-b4 Be7xb4 36.Rb2xb4 Rf5-f4 37.Rf1-d1 Qc6-c8 38.Kh2-g1 Qc8-f8 39.Rd1-b1 Rf4xd4 40.Rb4xb5 Rb8xb5 41.Rb1xb5 g5-g4 42.Bh5xg4 Rd4-d3 43.Bg4-f3 Bd5xf3 44.g2xf3 Rd3-b3 45.Rb5-b6 Rb3xb6 46.a5xb6 c4-c3 47.Qg3-h4 Ka8-b7 48.Qh4-a4 Qf8-c5 49.Qa4-a7+ Kb7-c6 50.Qa7-c7+........... The position is equal: = (0.00) Depth: 53/47 00:02:17 7056MN, tb=8050284 33.Rc2-b2 Rf8-b8 34.Rf1-b1 Kb7-a8 35.Bh5-e2 Bd5-e4 36.Rb1-e1 Be4-d5 The position is equal: = (0.00) Depth: 54/65 00:04:10 12985MN, tb=14949214 33.Rc2-b2 Rf8-b8 34.Rf1-b1 Kb7-a8 35.Bh5-e2 Bd5-e4 36.Rb1-c1 Be4-d5 The position is equal: = (0.00) Depth: 55/9 00:04:23 14313MN, tb=15835853

BTW what's up with Depth 55/9 ? I have seen that before but seems strange.

Alayan-stk-2 commented 4 years ago

The low seldepth has been seen in a game blundered at CCC. I have a hard time believing the seldepth reporting is legit in those situations, but in that other case SF expected a quick transition to TB that was incorrect (one of the expected moves was a blunder).

vondele commented 4 years ago

the low seldepth can happen IMO. I'm not sure this is the real issue.

NKONSTANTAKIS commented 4 years ago

The cycle detection mechanism just came to mind. By a-priori detecting no-progress via transposition, duplicate search is avoided, but what happens when the few in-between moves alter a 50-move win to a 50-move draw or a draw to a loss? Pretty rare, since 3 things need to coincide:

  1. cycle detection to trigger inside the tree
  2. the searched objective game result to alter in-between the triggering
  3. the selected move, via this misinformation, to unluckily blunder the real outcome

Pure speculation here, since cycle detection was AFAIR introduced just before SF9.

Nvm it was just after SF9 https://github.com/official-stockfish/Stockfish/commit/91a76331ca27b40d63f0031fbd7b9e41ead354d4 So basically sorry for noise, & hopefully some imaginative solution at that GHI-ish area.