Open robertnurnberg opened 2 months ago
For completeness, the output to the final go nodes
command is
info depth 1 seldepth 8 multipv 1 score cp 81 nodes 78 nps 382 hashfull 0 tbhits 0 time 204 pv f6e8
info depth 2 seldepth 3 multipv 1 score cp 81 nodes 130 nps 637 hashfull 0 tbhits 0 time 204 pv f6e8
info depth 3 seldepth 3 multipv 1 score cp 81 nodes 180 nps 882 hashfull 0 tbhits 0 time 204 pv f6e8
info depth 4 seldepth 3 multipv 1 score cp 81 nodes 231 nps 1132 hashfull 0 tbhits 0 time 204 pv f6e8
info depth 5 seldepth 4 multipv 1 score cp 84 nodes 286 nps 1401 hashfull 0 tbhits 0 time 204 pv f6e8 h4e1
info depth 6 seldepth 5 multipv 1 score cp 80 nodes 466 nps 2284 hashfull 0 tbhits 0 time 204 pv f6e8 h4e1 f7f5 e1g3
info depth 7 seldepth 10 multipv 1 score cp 78 nodes 768 nps 3764 hashfull 0 tbhits 0 time 204 pv f6e8 h4e1 f7f5 c3e2 e8d6
info depth 8 seldepth 12 multipv 1 score cp 69 nodes 3066 nps 14669 hashfull 1 tbhits 0 time 209 pv f6e8 h4e1 f7f5 a3a4 a5a7 b2c1
info depth 9 seldepth 13 multipv 1 score cp 69 nodes 6902 nps 32102 hashfull 2 tbhits 0 time 215 pv e7c5 h4b4 f6d5 b4c5 a5c5
info depth 10 seldepth 15 multipv 1 score cp 69 nodes 9301 nps 42277 hashfull 4 tbhits 0 time 220 pv f6e8 h4b4 e7c7 c3e4 a5b5
info depth 11 seldepth 16 multipv 1 score cp 71 nodes 15718 nps 67750 hashfull 7 tbhits 0 time 232 pv f6e8 h4e1 f7f5 a3a4 g7f6 e1c1 e8d6
info depth 12 seldepth 11 multipv 1 score cp 72 nodes 18540 nps 78227 hashfull 8 tbhits 0 time 237 pv f6e8 h4e1 f7f5 a3a4 g7f6 e1c1 e8d6 b2a3 e5e4
info depth 13 seldepth 17 multipv 1 score cp 73 nodes 31013 nps 117030 hashfull 11 tbhits 0 time 265 pv f6e8 h4e1 f7f5 a3a4 g8h7 b2c1 a5a7 e3e4 f5f4 c1d2
info depth 14 seldepth 20 multipv 1 score cp 77 nodes 50784 nps 165420 hashfull 18 tbhits 0 time 307 pv f6e8 h4e1 f7f5 a3a4 g8h7 b2c1 a5a7 c1d2 e8d6 a4a5 e5e4 d3e4 e6c4 e4f5 c4f1
info depth 15 seldepth 20 multipv 1 score cp 80 nodes 105665 nps 257719 hashfull 40 tbhits 0 time 410 pv f6e8 h4e1 e8d6 a3a4 d6f5 b2c1 h5h4 h1g1 f5g3 f1f2 f7f5 c1d2 a5a7 e3e4 f5f4
info depth 16 seldepth 26 multipv 1 score cp 71 nodes 235967 nps 371017 hashfull 99 tbhits 0 time 636 pv f6e8 h4e1 e8d6 b2c1 f7f5 e3e4 f5f4 a3a4 d6f7 c1d2 e7c7 e1h4 c7c8 h4e1 a8a6
info depth 17 seldepth 25 multipv 1 score cp 74 nodes 343756 nps 418704 hashfull 148 tbhits 0 time 821 pv f6e8 h4e1 e8d6 e3e4 b7b5 e1g3 d6b7 b2c1 b7c5 b1a1 a5a6
info depth 18 seldepth 25 multipv 1 score cp 74 nodes 400287 nps 434150 hashfull 176 tbhits 0 time 922 pv f6e8 h4e1 e8d6 e3e4 b7b5 e1g3 d6b7 b2c1 b7c5 b1a1 a5a6
bestmove f6e8 ponder h4e1
This appears to be a hash collision issue. I can't replicate with 64-bit tt key (same number of tt entries)
Did you use the script, or just the one reproduction? If the latter isn't it possible that the collision is responsible for the divergence but not directly the bad move?
not sure what you mean by "script"? I followed the steps to reproduce
He made a script to search for a reproduction.
See also here for a script to find the necessary commands to reproduce the "bug".
ok, I'll try to investigate further tomorrow
There is no guarantee for the script to find the right parameters (needed to trigger the bug) in finite time....
I was a bit lucky the first time round, and could use the knowledge at roughly what nodes count the move was produced during the LTC fishtest game.
I believe the best course of action would be do definitely pin down the cause, and then try to find a remedy.
Btw, the issue still triggers with setoption name hash value 147
. Running with that option should mean that we reduce the chances of general hash collisions, and so it might be easier to find the one responsible for the blunder (if indeed a hash collision is to blame).
Edit: Also the final command can be shortened to go nodes 1
to still trigger the blunder as best move.
I figured a smaller reproduction and there is only 6 collided positions, from the first glance it looks they are irrelevant smaller repro for sf revision https://github.com/official-stockfish/Stockfish/commit/0716b845fdef8a20102b07eaec074b8da8162523
position fen r1bq1rk1/pppn1pbp/3p1np1/4P3/1P6/P3PN2/1BPP2PP/RN1QKB1R b KQ - 0 8
go nodes 109876
position fen r4rk1/ppp2pbp/3q1np1/4pb2/1P6/P1N1P3/1BPPB1PP/R2Q1RK1 b - - 5 12
go nodes 65986
position fen r4rk1/ppp2pb1/3q1np1/4pb1p/1P6/P1NPP3/1BP1B1PP/R2Q1RK1 b - - 0 13
go nodes 21611
position fen r5k1/1p2qpb1/2p1bnp1/r3p2p/7Q/P1NPPB1P/1BP3P1/1R3R1K b - - 2 20
go nodes 1
ucinewgame
results
posKey: 156379583
FENs:
Number of FENS: 2
r1b1nrk1/pppn1pbp/8/6p1/1P1Np2q/P1N1P1P1/1BPPB2P/R2QK2R w KQ - 0 13
4r1k1/1pp2pb1/rq3np1/4pb1p/4P3/P1NP4/1BP1B1PP/R3QR1K w - - 3 18
TT Moves: (none)
(none)
(none)
(none)
(none)
posKey: 1476590525
FENs:
Number of FENS: 2
r2q1rk1/1pp2pbp/5np1/p3p3/1P6/P1N1P3/1BbPB1PP/R2Q1RK1 w - - 0 13
4r1k1/2p2p2/1p3npb/r1q2b1p/4B3/P1NPP2P/2P3P1/1RB1QRK1 b - - 0 21
TT Moves: (none)
(none)
posKey: -391743011
FENs:
Number of FENS: 2
r1b2rk1/1ppnqpb1/5n2/p5pp/1P1Np3/P1N1P3/1BPPB1PP/R2Q1R1K w - - 0 14
r4rk1/pppn1pbp/2q3p1/4p3/1P6/P1NPP3/1B1P2PP/R2Q1RK1 w - - 1 15
TT Moves: (none)
(none)
posKey: 424872270
FENs:
Number of FENS: 2
5r2/ppp1qpkp/6P1/8/1P6/P3P3/2P1B1PP/R2r1RK1 b - - 0 18
3r1rk1/ppp2p2/3q2p1/7p/1P2P3/P3P3/2P2KPb/1R1BQR2 b - - 1 20
TT Moves: (none)
(none)
posKey: -457919305
FENs:
Number of FENS: 2
3rr1k1/ppp2pb1/3q2p1/3n1b1p/1P1Pp3/P1N1P3/1BP1B1PP/R3QR1K w - - 4 17
3r1rk1/ppp2pbp/5np1/4pb2/1P2P3/P1N5/1BP1B1PP/2R2RK1 b - - 0 15
TT Moves: (none)
(none)
posKey: -1351134240
FENs:
Number of FENS: 2
6k1/1p3p2/2p1b1pb/2Bnp2p/r3N3/3PPB1P/2P3P1/6RK b - - 3 26
r5k1/5pb1/4bnp1/rp1pQ2p/8/P2PqB1P/1BP3P1/1R3R1K b - - 0 22
TT Moves: (none)
(none)
After thorough analysis, I have found the culprit which is this position (the position after the punishment of the blunder) of r3n1k1/1p2Qpb1/2p1b1p1/r3p2p/8/P1NPPB1P/1BP3P1/1R3R1K b
is never evaluated and is never hit in search, the only hits are at qsearch and we immediately cutoff with standpating based on a TT entry whereas we never saved this position ever into TT.
This proves that there was a hash collision that caused the queen's blunder.
here is the output for the collided positions using in qsearch and search for that position
if (uint16_t(posKey) == uint16_t(2069647028597739265uLL))
{
sync_cout << std::endl << pos.fen() << sync_endl;
}
From @peregrineshahin's output, these are the positions that collide in TT:
5rk1/1pp2pb1/5np1/4pb1p/1q2P3/2NP4/1BP1B1PP/Q4R1K w - - 0 17
5rk1/1pp2pb1/5np1/4pb1p/1q2P3/2NP4/1BP1B1PP/Q4R1K w - - 0 18
4r1k1/1pp2p2/5np1/2r2b1p/4p3/P1N1b3/2PQB1PP/1RB2R1K w - - 0 21
r3n1k1/1p2Qpb1/2p1b1p1/r3p2p/8/P1NPPB1P/1BP3P1/1R3R1K b - - 0 21
The first two are the same position with different full move counter, and the final one is the position after the blunder was punished. Both positions for white to move are clearly winning, the second one more so. I believe the latter one leads the incorrect (winning) eval for the blundered position.
Corrected: Only true collisions as per https://github.com/official-stockfish/Stockfish/issues/5174#issuecomment-2060953666 are
4r1k1/1pp2p2/5np1/2r2b1p/4p3/P1N1b3/2PQB1PP/1RB2R1K w - - 0 21
r3n1k1/1p2Qpb1/2p1b1p1/r3p2p/8/P1NPPB1P/1BP3P1/1R3R1K b - - 0 21
Yes worth noting that the value we cut off with is 1516 which is a winning eval for the side to move. this happened in qsearch stand. pat. (immediately cut off with the wrong tt value) i.e. the hash collision is in effect:
diff --git a/src/search.cpp b/src/search.cpp
index 24805aa7..5a1e993b 100644
--- a/src/search.cpp
+++ b/src/search.cpp
@@ -1445,6 +1445,8 @@ Value Search::Worker::qsearch(Position& pos, Stack* ss, Value alpha, Value beta,
{
if (ss->ttHit)
{
+ // Wrong ttHit as we never saved this position before
+
// Never assume anything about values stored in TT
unadjustedStaticEval = tte->eval();
if (unadjustedStaticEval == VALUE_NONE)
@@ -1456,6 +1458,9 @@ Value Search::Worker::qsearch(Position& pos, Stack* ss, Value alpha, Value beta,
if (ttValue != VALUE_NONE
&& (tte->bound() & (ttValue > bestValue ? BOUND_LOWER : BOUND_UPPER)))
bestValue = ttValue;
+
+ // best value is set to a wrong value of 1516
+ // for r3n1k1/1p2Qpb1/2p1b1p1/r3p2p/8/P1NPPB1P/1BP3P1/1R3R1K b - - 0 21
}
else
{
@@ -1470,10 +1475,12 @@ Value Search::Worker::qsearch(Position& pos, Stack* ss, Value alpha, Value beta,
// Stand pat. Return immediately if static value is at least beta
if (bestValue >= beta)
{
+ // ohuh we are about to Stand pat.
if (!ss->ttHit)
tte->save(posKey, value_to_tt(bestValue, ss->ply), false, BOUND_LOWER, DEPTH_NONE,
Move::none(), unadjustedStaticEval, tt.generation());
+ // wrong bestValue is immediatly used and cutoff happened
return bestValue;
}
For completeness, it looks like there was a missing check in my analysis that led to identifying the only collision ( culprit )
which is with 4r1k1/1pp2p2/5np1/2r2b1p/4p3/P1N1b3/2PQB1PP/1RB2R1K w - - 0 21
Describe the issue
Stockfish playing as black hangs the queen in the position
r5k1/1p2qpb1/2p1bnp1/r3p2p/7Q/P1NPPB1P/1BP3P1/1R3R1K b - -
by suggesting as bestmovef6e8
Expected behavior
Stockfish should suggest any other available move, that does not blunder the position in a single move.
Steps to reproduce
These steps deterministically lead to the described issue. They only work for sf revision https://github.com/official-stockfish/Stockfish/commit/0716b845fdef8a20102b07eaec074b8da8162523
Anything else?
Losing games by hanging the queen in games on fishtest has been anecdotally reported by @dav1312 . The steps to reproduce this example blunder were initiated by his post on discord. See also here for a script to find the necessary commands to reproduce the "bug".
Operating system
All
Stockfish version
0716b845fdef8a20102b07eaec074b8da8162523