shuffle-detection patch still glitchy, can cause loss in drawn positions

[adentong]

From the forum:

May I draw attention to the discussion on http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=582810 The bottom line, as far as I understand it, is that SF now dislikes shuffling so much that sometimes it plays nonsensical captures etc in positions where shuffling to a draw would be best play. This patch has a history: included in the master, then reverted by Marco Costalba, only to be reinstated later in modified form. I contend that the modification fixed only one problem with the patch, but left another (potentially losing) fault in place.

Since the Rybkaforum thread is now a couple weeks old, I retested with SF 31 May 2019, in the position which I call example 1 in the forum thread: 3k4/1r2p3/r2pPp2/b1pP1Pp1/1pP3Pp/pP5P/P3K3/8 b - - 0 6

This is a dead draw as long as White declines any rooks offered as bait. Indeed, pre-patch SF has no problems at all, playing the position correctly and displaying sound PVs, albeit with a faulty high eval for Black. But Stockfish_19053114_x64_modern does as shown below: it actually plans to lose this easy draw, it uses tons of time to reach an unimpressive depth, and the eval didn't get any better; just broken. This was on 28 threads with 96 GB hash; the issue may be less likely to occur on a smaller PC, but it's real nonetheless.

Thanks for any follow-up, I'd be very happy to have a Fish with working shuffle awareness. Meanwhile, does somebody know how to get hold of the source for say SF 18 Apr 2019, one of the last ones that still was OK. I can then hopefully make a custom N-move rule with N<50, which should largely cover the shuffle issue in a much simpler way.

FEN: 3k4/1r2p3/r2pPp2/b1pP1Pp1/1pP3Pp/pP5P/P3K3/8 b - - 0 6

Stockfish_19053114_x64_modern: 1/1 00:00 2k 1,191k +17.37 Kc7 2/2 00:00 4k 2,026k +17.51 Kc7 Kf2 3/3 00:00 6k 3,221k +17.57 Kc8 Kd3 Bb6 4/4 00:00 10k 4,917k +17.66 Bb6 Ke3 Kc8 Kf2 5/5 00:00 11k 5,485k +17.66 Bb6 Kf3 Kc7 Kf2 Rb8 6/6 00:00 13k 6,394k +17.66 Bb6 Ke3 Kc8 Kf2 Kc7 Kf1 7/7 00:00 26k 8,813k +17.74 Kc7 Kf3 Rb8 Kg2 Bb6 Kh2 8/8 00:00 39k 13,166k +17.75 Kc8 Kf3 Rb8 Kf2 Bb6 Kf1 Kc7 Kg1 9/9 00:00 43k 14,394k +17.76 Kc8 Kf3 Bb6 Kg2 Rb8 Kf2 Kb7 Kg2 Rd8 10/10 00:00 52k 17,204k +17.76 Kc8 Kf2 Rb8 Kg1 Kc7 Kg2 Bb6 Kg1 Re8 Kg2 11/11 00:00 86k 21,414k +17.77 Kc8 Kf2 Bb6 Ke1 Rc7 Kf1 Kb7 Kf2 Ra8 Kg2 Re8 12/13 00:00 110k 27,489k +17.77 Kc8 Kf2 Rb8 Kg1 Kc7 Kf2 Bb6 Kg2 Rba8 Kf2 R8a7 Kf3 Kc8 13/13 00:00 159k 31,807k +17.82 Kc8 Kf2 Rc7 Kg2 Bb6 Kf2 Kb7 Kf1 Ra8 Kg1 Re8 Kg2 Rcc8 14/14 00:00 192k 31,944k +17.76 Kc8 Kf2 Bb6 Kg2 Rc7 Kf2 Kb7 Kf1 Ra8 Kf2 Re8 Kg1 Rcc8 Kg2 15/15 00:00 292k 36,442k +17.75 Kc8 Kf2 Rc7 Kg2 Rca7 Kh1 Bb6 Kg2 Rb7 Kf1 Rc7 Kg2 Kb7 Kf1 Ra8 16/17 00:00 345k 43,157k +17.75 Kc8 Kf1 Kd8 Kg2 Bb6 Kg1 Ke8 Kf1 Ra8 Kg1 Kf8 Kf2 Kg7 Kg1 Rba7 Kg2 17/17 00:00 372k 41,343k +17.75 Kc8 Ke1 Rc7 Kf2 Bb6 Kg1 Kd8 Kg2 Rb7 Kg1 Ke8 Kh1 Rb8 Kg1 Kf8 Kf1 Kg7 18/18 00:00 430k 42,991k +17.75 Kc8 Ke1 Rc7 Kf2 Bb6 Kg1 Kd8 Kg2 Ke8 Kg1 Kf8 Kf2 Kg7 Kg2 Kg8 Kg1 Rb7 Kf2 19/57 00:00 786k 49,149k +17.75 Kc8 Kf3 Bb6 Kf2 Kd8 Kf3 Rb8 Kg2 Ke8 Kg1 Kf8 Kf1 Rb7 Kg2 Ba5 Kf2 Kg7 Kg1 20/62 00:00 1,136k 51,622k +17.66 Kc8 Kf1 Ra8 Kg1 Bb6 Kf2 Rc7 Kg2 Ra6 Kf2 Ba5 Kg1 Kd8 Kf1 Bb6 Kg2 Rb7 Kf3 Rb8 21/66 00:00 3,833k 52,500k +17.37 Kc8 Kf1 Rba7 Kg2 Kd8 Kg1 Ra8 Kg2 Bb6 Kg1 Kc8 Kf2 Ba5 Kg1 Kc7 Kf2 Rd8 Kg1 Ra7 Kf2 Rb7 Kf1 Ra8 Kg1 Ra6 Kg2 Kd8 Kf1 Bb6 Kg1 Ke8 Kg2 Bc7 Kf1 Ra5 Kg1 Raa7 Kg2 Ba5 Kf2 Ra8 Kg2 Rc7 Kf1 Rca7 Kg1 Rb8 Kg2 Bb6 Kf2 Rab7 Kg2 Kd8 Kf2 Rc7 Kg1 Rbb7 Kf2 Ke8 Ke1 Ra7 Kf2 Ra8 Kg2 Kf8 22/68 00:00 4,163k 52,694k +17.37 Kc8 Kf1 Rba7 Kg2 Kd8 Kg1 Ra8 Kg2 Bb6 Kg1 Kc8 Kf2 Ba5 Kf1 Kc7 Kf2 R8a7 Kg2 Kc8 Kg1 Kd8 Kh1 Ke8 Kg1 Rb7 Kf1 Kf8 Kf2 Ra8 Kg1 Bc7 Kg2 Bb6 Kf2 Rba7 Kg2 Bd8 Kf2 Ba5 Kg2 Bb6 Kf2 Ra6 Kg2 Bd8 Kf2 Ba5 Kg2 Rb8 Kg1 Ke8 Kf2 Kd8 Kf1 Raa8 Kg2 Kc8 Kf2 Ra7 Kg1 Rab7 Kf2 Rc7 Kg2 Ra8 Kf2 Bb6 Kg2 23/71 00:00 5,926k 52,911k +17.27 Kc8 Kf1 Ra8 Kg1 Kd8 Kg2 Bb6 Kg1 Ke8 Kf1 Bd8 Kg2 Rbb8 Kg1 Bb6 Kg2 Kd8 Kf2 Ra7 Kf3 Rab7 Kf2 Kc8 Kf1 Kc7 Kg1 Rh8 Kg2 Ba5 Kf1 Ra8 Kg2 Kb6 Kg1 Ka6 Kg2 Bc7 Kf2 Ka5 Kf1 Rbb8 Kg2 Re8 Kf2 Ra7 Kg2 Rea8 Kf1 Ka6 Kg1 Kb7 Kf2 Bd8 Kg2 Ba5 Kg1 Rb8 Kf2 Kb6 Kf1 Ra6 Kf2 Raa8 Kg2 Kb7 Kg1 Bb6 Kf2 Re8 Kf1 Kc7 24/74 00:00 6,068k 52,768k +17.27 Kc8 Kf1 Ra8 Kg1 Rba7 Kf1 Bd8 Kg2 Ra6 Kg1 Ba5 Kf1 Kc7 Kf2 Rb8 Kg2 Kb6 Kf2 Kb7 Kg1 Bb6 Kg2 Raa8 Kf2 Kc7 Kg2 Kd8 Kf2 Ke8 Kg2 Kf8 Kf2 Kg7 Kg2 Ra5 Kf2 Ra6 Kf1 Rh8 Kf2 Rha8 Kg1 Kf8 Kg2 Rb8 Kg1 Rb7 Kg2 Kg7 Kf2 Ba5 Kg1 Bc7 Kf2 Bb6 Kg2 Raa7 Kf1 Ba5 Kg1 Kf8 Kf1 Bd8 Kg1 Bc7 Kf2 Rb8 Kf1 Raa8 Kg1 Ba5 Kg2 Ra6 Kf1 25/77 00:00 9,115k 52,689k +17.18 Kc8 Kf1 Kc7 Kg2 Rb8 Kf2 Kc8 Kf1 Ra7 Kg1 Raa8 Kg2 Bb6 Kg1 Ra6 Kf1 Ra7 Kf2 Kc7 Kg2 Kd8 Kg1 Rbb7 Kf2 Bc7 Kg2 Ra6 Kh1 Rb8 Kg2 Ra5 Kf2 Ke8 Kg2 Bb6 Kf2 Raa8 Kg1 Ra6 Kf2 Kf8 Kg1 Ba5 Kf1 Kg7 Kg1 Rh8 Kf1 Ra7 Kf2 Raa8 Kf1 Kf8 Kg2 Ke8 Kf1 Kd8 Kg2 Kc8 Kf2 Kc7 Kf1 Rhe8 Kg2 Kb6 Kg1 Ra7 Kg2 Rb8 Kg1 Rba8 Kg2 Kb7 Kg1 Ra6 Kg2 R8a7 26/80 00:00 9,547k 52,743k +17.18 Kc8 Kf1 Kc7 Kg2 Rb8 Kf2 Kc8 Kf1 Raa8 Kg2 Bb6 Kg1 Ra6 Kf2 Ra7 Kf1 Kc7 Kf2 Raa8 Kg2 Rb7 Kf1 Rab8 Kg2 Ba5 Kf2 Kc8 Kf1 Bc7 Kf2 Kd8 Kg1 Ra8 Kf1 Ke8 Kf2 Ba5 Kg2 Bb6 Kf2 Rab8 Kg1 Rc7 Kf1 Ra8 Kg1 Ba5 Kg2 Kf8 Kg1 Rb8 Kf2 Kg7 Kg2 Ra8 Kf1 Bb6 Kg1 Rb8 Kf1 Rh8 Kg2 Ba7 Kf2 Rb7 Kg2 Rbb8 Kf2 Ra8 Kf3 Bb6 Kf2 Ba5 Kg1 Ra6 Kf2 Rb8 Kf1 Kf8 Kg1 27/83 00:00 9,897k 52,641k +17.18 Kc8 Kf1 Kc7 Kg2 Rb8 Kf2 Kc8 Kg2 Bb6 Kf2 Ra7 Kf1 Kc7 Kf2 Rbb7 Kg1 Kc8 Kf1 Rb8 Kg2 Rab7 Kf1 Ba5 Kf2 Rc7 Kf1 Rbb7 Kg1 Kd8 Kg2 Ke8 Kf2 Rb8 Kg2 Rcb7 Kf1 Ra7 Kg1 Kf8 Kf2 Ra6 Kf1 Rba8 Kg1 Bb6 Kg2 Kg7 Kf1 Rh8 Kg2 Rb8 Kf2 Ba5 Kf1 Raa8 Kg2 Rc8 Kf1 Rh8 Kg2 Ra7 Kf1 Bb6 Kg2 Bc7 Kf1 Rha8 Kg1 Bb6 Kf1 Rc7 Kg1 Rb7 Kg2 Kf8 Kg1 Rba7 Kg2 Ra6 Kf1 Bc7 Kf2 R8a7 28/86 00:00 10,598k 52,466k +17.18 Kc8 Kf1 Kc7 Kg2 Rb8 Kg1 Bb6 Kg2 Kd8 Kg1 Kc8 Kf2 Ba5 Kg2 Kd8 Kg1 Raa8 Kf2 Bc7 Kf1 Kc8 Kg2 Rb7 Kf1 Rab8 Kf2 Ba5 Kf1 Kd8 Kg2 Ke8 Kf1 Ra7 Kg1 Bb6 Kf2 Kf8 Kf1 Bc7 Kg2 Raa8 Kf1 Ra5 Kf2 Ke8 Kg2 Rba8 Kg1 R5a7 Kf2 Ba5 Kg1 Rb7 Kf2 Kf8 Kg2 Bb6 Kg1 Rba7 Kg2 Ra6 Kf1 Ba5 Kf2 Rb8 Kf1 Rb7 Kg1 Rc7 Kf1 Raa7 Kg2 Rab7 Kf2 Ke8 Kg2 Kd8 Kg1 Bb6 Kh1 Ra7 Kg1 Kc8 Kf2 Ba5 Kg2 Ra8 29/89 00:00 10,790k 52,635k +17.18 Kc8 Kf1 Bb6 Kg1 Ra8 Kf2 Raa7 Kg2 Kd8 Kf2 Ba5 Kg1 Ke8 Kf2 Rb8 Kg1 Bb6 Kf2 Kf8 Kf1 Ba5 Kg1 Raa8 Kf2 Ke8 Kf1 Ra6 Kf2 Bd8 Kf1 Bb6 Kf2 Rba8 Kg2 Kd8 Kg1 Rb8 Kg2 Raa8 Kf1 Ra7 Kf2 Rba8 Kg2 Bc7 Kg1 Kc8 Kf2 Ra5 Kg2 Bb6 Kf2 Kc7 Kg1 Rh8 Kg2 Rb8 Kg1 Raa8 Kf1 Rh8 Kg2 Ba5 Kf1 Kd8 Kg2 Rf8 Kf2 Ke8 Kf1 Rh8 Kg2 Ra7 Kf1 Kd8 Kg1 Bb6 Kf1 Ra8 Kf2 Kc8 Kf1 Rh7 Kg1 Rh6 Kf2 Kc7 Kg1 Ba5 30/92 00:00 11,363k 52,608k +17.18 Kc8 Kf1 Bb6 Kg1 Ra8 Kf2 Ra5 Kg1 Rb8 Kg2 Raa8 Kf1 Ra6 Kf2 Ba5 Kg1 Raa8 Kg2 Ra7 Kf1 Rba8 Kg2 Kb7 Kg1 Ra6 Kf1 Rh8 Kf2 Rb6 Kg1 Kc7 Kf1 Ra8 Kg2 Rba6 Kg1 Rh8 Kf2 Bb6 Kg1 Ra5 Kf1 Raa8 Kg2 Kd8 Kf2 Ra5 Kg1 Bc7 Kf2 Ra7 Kf1 Ke8 Kf2 Kf8 Kg2 Kg7 Kf1 Rb8 Kg1 Raa8 Kf1 Kg8 Kf2 Bb6 Kg2 Rb7 Kf1 Kf8 Kg1 Rc7 Kf2 Raa7 Ke2 Rcb7 Kf2 Kg7 Kg1 Rc7 Kg2 Rab7 Kf1 Kf8 Kf2 Ba5 Kg2 Rb8 Kf2 Ke8 Kg1 Ra7 Kf2 31/95 00:00 12,683k 52,848k +17.18 Kc8 Kf1 Bb6 Kg1 Rc7 Kg2 Ba5 Kg1 Ra8 Kf2 Kd8 Kf1 Rb8 Kf2 Rcb7 Kg1 Ke8 Kg2 Bb6 Kg1 Ra7 Kg2 Rba8 Kf1 Kd8 Kg2 Ra6 Kf1 Bc7 Kg1 R6a7 Kg2 Kc8 Kf2 Ra5 Kg1 Rb8 Kf2 Bb6 Kf1 Rba8 Kg2 R8a6 Kf2 Kb7 Kg1 Ra7 Kf1 Bc7 Kf2 R5a6 Kg1 Ra8 Kg2 Bb6 Kg1 Kc8 Kf2 Rb8 Kg1 Kc7 Kg2 Rb7 Kf1 Raa7 Kg1 Ra8 Kg2 Rh8 Kg1 Rh6 Kf2 Ba5 Kf1 Rb6 Kg2 Ra6 Kf1 Rh8 Kg1 Rha8 Kf2 Kb7 Kg2 Rb6 Kf2 Ra7 Kg2 Kc8 Kf2 Kd8 Kg2 Rbb7 Kg1 Ke8 32/98 00:00 13,051k 52,840k +17.18 Kc8 Kf1 Rba7 Kf2 Kd8 Kg1 Ke8 Kg2 Bd8 Kf1 Rc7 Kf2 Ra8 Kf1 Rb7 Ke2 Ra6 Ke1 Ba5 Kf2 Raa7 Kf1 Rb8 Kg2 Bb6 Kf2 Kd8 Kf1 Rab7 Kg2 Ke8 Kg1 Ba5 Kg2 Ra7 Kg1 Raa8 Kf1 Bb6 Kf2 Kf8 Kf1 Rc8 Kf2 Ra5 Kg2 Rc7 Kg1 Rca7 Kf2 Rb7 Kg1 Rc7 Kg2 Ke8 Kg1 Kd8 Kg2 Rca7 Kg1 Ra8 Kf2 R8a6 Kg2 Kc8 Kf1 Kb7 Kg1 Ra7 Kf1 Bc7 Kf2 R5a6 Kf1 Ba5 Kf2 Bb6 Kg2 Kc7 Kg1 Rb7 Kf2 Rb8 Kg2 Raa8 Kf2 Kc8 Kf1 Ba5 Kf2 Kb7 Kg1 Bb6 Kg2 Rh8 Kf2 Ra6 Kg2 33/101- 01:04 2,364,639k 36,645k +17.08 Kc8 Kf2 33/102+ 08:29 17,071,033k 33,497k +17.18 Kc8 33/102 10:31 21,092,318k 33,424k +17.09 Kc8 Kd1 Rb8 Kc1 Raa8 Kd2 Ra7 Ke3 Bb6 Ke2 Ra6 Kf2 Kd8 Ke2 Ke8 Kf2 Ra4 Kg2 Ra7 Kf2 Rba8 Kg2 Bc7 Kf3 Ra4 bxa4 Rxa4 Kf2 Kd8 Kg1 Kc8 Kg2 b3 34/102 17:23 34,809,053k 33,363k +17.09 Kc8 Kd1 Rb8 Kc1 Raa8 Kd2 Bc7 Ke2 Bb6 Ke3 Kc7 Ke2 Kd8 Kf2 Rb7 Ke1 Rc8 Kf2 Ke8 Ke3 Ra8 Kd2 Raa7 Ke3 Ra4 Kf2 Kf8 bxa4 Ke8 a5 Bxa5 Kf3 b3 Kg2 35/102 31:08 61,313,817k 32,808k +17.09 Kc8 Kd1 Rb8 Kc1 Raa8 Kd2 Bc7 Ke2 Bb6 Ke1 Ra7 Kd2 Rab7 Ke3 Kd8 Kf2 Ke8 Kg2 Ra8 Kf2 Kf8 Kg1 Rc8 Kg2 Bc7 Kf1 Rcb8 Kf2 Ke8 Kg2 Ra7 Kf3 Ba5 Ke2 Bb6 Kd1 Ra5 Kc2 Ra6 Kd2 Kf8 Kc2 Rc8 Kd2 Raa8 Ke2 Ra7 Kf1 Ra6 Kg2 Ba5 Kf3 Bd8 Ke3 Kg8 Ke2 Bb6 Kd1 Ra4 bxa4 b3 axb3 36/102 38:50 75,631,495k 32,451k +17.09 Kc8 Kd1 Rb8 Kc1 Raa8 Kd2 Bc7 Ke2 Bb6 Ke1 Bd8 Ke2 Ra7 Kd2 Rab7 Kd3 Bc7 Ke2 Ba5 Kd2 Bb6 Ke3 Ra8 Ke2 Rba7 Kd1 Rc7 Ke2 Ba5 Kf2 Raa7 Ke2 Ra6 Kd3 Bb6 Ke2 Kd8 Ke3 Rca7 Ke2 Ra5 Kf2 Ke8 Ke2 Bc7 Ke1 Kf8 Kd2 Kg8 Ke2 R7a6 Kd1 Ra8 Kd2 Kg7 Kc1 Kf8 Kd1 Bb6 Kc2 R5a7 Kd2 Rb8 Kd1 Ke8 Kc1 Ra4 bxa4 b3 axb3

[adentong]

Since this seems like a legit issue, I'm opening an issue here since it'll probably be viewed and discussed more than on the forum.

[Rocky640]

Here is a link to the patch which was committed just before the first shuffle patch https://github.com/official-stockfish/Stockfish/tree/6373fd56e90bc6114230a70cacad804248d955e2

[MJZ1977]

@adentong : I don't understand how SF is losing in that position. Because it hangs and lose on time ?

The problem is not the shuffle patch, but probably the simplification that came after it and removes one part of the patch. This simplification seems OK for most of the "normal positions", it can only leads to long hangs.

[adentong]

@MJZ1977 I actually copy and pasted the post from fishcooking since I thought more people would come across it here. But the original poster seems to be implying that in the position mentioned post-shuffling patch it would make the wrong move and lose an otherwise drawn game.

[adentong]

@MJZ1977 Relaying the original poster's response:

I've seen people are starting to take note on Github. Since I don't have an account there, let me reply here to MJZ1977 in the hope that it somehow finds its way. Look at the last three plies in, say, the final d=36 PV which I copied: SF, playing as White, plans to take a black rook on a4, enabling the black b3, opening the position. After this, Black will easily mate with his remaining material. If you look at the original Rybkaforum thread, MrKris confirmed that this isn't just some PV bug. When he let the position be autoplayed, SF actually played these moves on the board, after which it was quickly game over of course. Seems Stockfish took "shuffling is bad" a bit too literally-typical computer. :-)

I see that VdV @vondele is running something shuffle-related on Fishtest; not sure if that is already a candidate fix. I repeat my question if the extensions are really essential to the idea: wouldn't it be enough to just penalize lines which end in long shuffle sequences? Anyway, the active devs will hopefully have the best idea about this.

[MJZ1977]

@adentong : I don't think that taking a rook on a4 have anything with shuffle patch. But I can be wrong. SF take it because in alpha-beta it thinks it is the better move at low depths and did'nt see that black can't break the fortress. The shuffle patch goal is to correct this assumption and detecte shuffling positions. But indeed, in your position it did'nt detect it and the original problem is still there. Anyway, I think that shuffle patch have been simplified and lost a part of its original idea. Please also note that these positions are so strange that in fishtest we never had them. For example, in original patch there is a limitation of shuffle extension after ply 18. It has been simplified and removed. In my original idea it was a security against strange positions that oblige SF to search low depths and tactical lines before going to extensions. I don't know if you can do it, but if you add this condition do you still have the problem? && ss->ply > 18

[adentong]

@MJZ1977 I refer you to the original thread on fishcooking. https://groups.google.com/forum/?fromgroups=#!topic/fishcooking/ZkejZXLInWY

[syzygy1]

The Rybka thread is very old.

The worst thing that could possibly happen (if at all) with the current shuffle extension is a search explosion that would prevent SF from reaching normal depths. If you see some other problem, then it's not the shuffle extension. (Search explosions are not supposed to happen anymore either, but I have not verified this myself.)

With the first versions of the shuffle extension, the search was adjucating certain positions in the middle of the search as a draw for no good reason. That certainly must have caused problems, but it was removed with @vondele's simplification (and, in my view, clear bug fix) of May 2: https://github.com/official-stockfish/Stockfish/commit/5c4002aa827653a125130a0d01d0bb96dd2b8bae

[syzygy1]

OK, this position does cause a search overflow. If you disable the shuffle extension, the search quickly gets to depth 64 and more. With the shuffle extension, it gets stuck (on my PC with 4GB hash and 6 threads) at depth 32.

So the logic limiting the extension is not yet sound. Well, unless this is exactly what is supposed to happen in this shuffle position, but there is no draw score being returned (maybe if I wait long enough?).

[syzygy1]

I guess the problem is that SF now accepts a rook sac in this position because due to the shuffle extension it can't see deep enough to see that accepting the sac is losing.

8/1r1rp1k1/1b1pPp2/2pP1Pp1/1pP3Pp/pP5P/P5K1/8 w - - 79 46

With 4GB and 6 threads, the search gets stuck at depth 33 with exd7 as the preferred move.

Without the extenion, SF switches from exd7 to Kf2 after 3 seconds (starting with an empty hash).

To disable the extension, add "&& 0" to the following line: https://github.com/official-stockfish/Stockfish/blob/46ce245763705c89dba60dcfda549dc1f64eb48b/src/search.cpp#L932

[syzygy1]

Reverting @vondele's simplification/bug fix may or may not "fix" this particular position (I have not tried it) but by almost certainly re-introducing other problems. At least I can see no justification for adjudicating positions in the middle of the tree as DRAW just because 15 or 18 moves have passed without pawn move or capture. It will just allow the engine to hide a loss behind paths that are just long enough to reach the 15 or 18 moves and discover the loss only when it's too late.

(I admit I have not yet tried to understand the full implication of the "ttHit && tte->depth() > depth" condition. But this obviously cannot make it safe in the case where the TT entry despite its higher depth does not cause a regular cutoff. And if it does cause a regular cutoff, why return VALUE_DRAW instead of the TT score??)

[vdbergh]

I guess the problem is that SF now accepts a rook sac in this position because due to the shuffle extension it can't see deep enough to see that accepting the sac is losing.

8/1r1rp1k1/1b1pPp2/2pP1Pp1/1pP3Pp/pP5P/P5K1/8 w - - 79 46

With 4GB and 6 threads, the search gets stuck at depth 33 with exd7 as the preferred move.

Without the extenion, SF switches from exd7 to Kf2 after 3 seconds (starting with an empty hash).

To disable the extension, add "&& 0" to the following line: https://github.com/official-stockfish/Stockfish/blob/46ce245763705c89dba60dcfda549dc1f64eb48b/src/search.cpp#L932

Strangely, with 1 thread there appears to be no problem in this position.

[syzygy1]

Strangely, with 1 thread there appears to be no problem in this position.

With 4 GB hash and 1 thread there does seem to be a problem.

info depth 33 seldepth 101 multipv 1 score cp -1225 nodes 4641351 nps 3448254 hashfull 0 tbhits 0 time 1346 pv e6d7 b7d7 g2f1 g7f7 f1f2 f7e8 f2g2 b6d8 g2g1 d7b7 g1f2 e8f8 f2g1 b7c7 g1f2 f8f7 f2g1 f7g8 g1f2 g8f8 f2g1 f8f7 g1g2 c7d7 g2f1 f7f8 f1g2 d7a7 g2f1 d8b6 f1g2 a7a6 g2f3 b6d8 f3e2 f8g8 e2e1 g8g7 e1e2 d8b6 e2f3 a6a7 f3e2 a7c7 e2f3 g7g8 f3g2 g8f7 g2f1 c7a7 f1g2 a7a8 g2f1 a8d8 f1e2 f7e8 e2f2 d8a8 f2f1 a8a7 f1f2 a7a6 f2f1 e8d7 f1f2 d7c7 f2g2 c7b8 g2g1 a6a5 g1g2 a5a7 g2f1 a7d7 f1f2 d7b7 f2g1 b6d8 g1f1 b8c8 f1g2 b7a7 g2f2 a7c7 f2f1 c7d7 f1g1 d8b6 g1f2 c8d8 f2e2 d8e8 e2f2 e8f7 f2e1 f7f8 e1e2 d7c7 e2f2 c7c8 f2g1

Here the engine gets stuck. seldepth = 101...

[syzygy1]

And I have the same problem with the default hash size in Stockfish.

[MichaelB7]

FWIW - and I could be wrong - but I believe this patch is directly responsible for solving this mate - in - 34 fairly quickly . If you remove it , the solution exploded to over 30 minutes on my machine. Wondering if somebody could also test - since I’m a little bit short of time to confirm. k1q4r/p7/P3P3/1N6/8/nQ2K3/6p1/2n5 w - - bm Qd5+; id "Arves.16248"; dm 34;

[vdbergh]

Strangely, with 1 thread there appears to be no problem in this position.

With 4 GB hash and 1 thread there does seem to be a problem.

info depth 33 seldepth 101 multipv 1 score cp -1225 nodes 4641351 nps 3448254 hashfull 0 tbhits 0 time 1346 pv e6d7 b7d7 g2f1 g7f7 f1f2 f7e8 f2g2 b6d8 g2g1 d7b7 g1f2 e8f8 f2g1 b7c7 g1f2 f8f7 f2g1 f7g8 g1f2 g8f8 f2g1 f8f7 g1g2 c7d7 g2f1 f7f8 f1g2 d7a7 g2f1 d8b6 f1g2 a7a6 g2f3 b6d8 f3e2 f8g8 e2e1 g8g7 e1e2 d8b6 e2f3 a6a7 f3e2 a7c7 e2f3 g7g8 f3g2 g8f7 g2f1 c7a7 f1g2 a7a8 g2f1 a8d8 f1e2 f7e8 e2f2 d8a8 f2f1 a8a7 f1f2 a7a6 f2f1 e8d7 f1f2 d7c7 f2g2 c7b8 g2g1 a6a5 g1g2 a5a7 g2f1 a7d7 f1f2 d7b7 f2g1 b6d8 g1f1 b8c8 f1g2 b7a7 g2f2 a7c7 f2f1 c7d7 f1g1 d8b6 g1f2 c8d8 f2e2 d8e8 e2f2 e8f7 f2e1 f7f8 e1e2 d7c7 e2f2 c7c8 f2g1

Here the engine gets stuck. seldepth = 101...

Ok. I had been testing with 64Mb.

[vondele]

The patch I'm testing does fine on a couple of these fens posted, if I read this issue correctly. See http://tests.stockfishchess.org/tests/view/5d03630d0ebc5925cf0a011a

[syzygy1]

FWIW - and I could be wrong - but I believe this patch is directly responsible for solving this mate - in - 34 fairly quickly . If you remove it , the solution exploded to over 30 minutes on my machine. Wondering if somebody could also test - since I’m a little bit short of time to confirm. k1q4r/p7/P3P3/1N6/8/nQ2K3/6p1/2n5 w - - bm Qd5+; id "Arves.16248"; dm 34;

The mate is found more quickly when I disable the shuffle extension.

[adentong]

I think I can close this now since the issue in the original post has been resolved by vondele's patch.