search

official-stockfish / Stockfish

A free and strong UCI chess engine
https://stockfishchess.org/
GNU General Public License v3.0
11.55k stars 2.27k forks source link

Stockfish hang in TCEC S16 Division P Game 51 #2291

Closed xoto10 closed 5 years ago

xoto10 commented 5 years ago

In the TCEC Archive here

Reproduces with a few seconds to move for me:

Engine master(1) failed to respond to ping
Terminating process of engine master(1)
Warning: Unexpected result claim from master: 1/2-1/2 {Draw by stalled connection}
Engine master(0) failed to respond to ping
Terminating process of engine master(0)
xoto10 commented 5 years ago

I used this as the opening book:

[FEN "8/4kp2/2Bp1p2/b1pP1P1p/P1P4P/6P1/8/6K1 b - - 19 57"]
[Result "1/2-1/2"]

1/2-1/2
vondele commented 5 years ago

can you give the precise sequence of uci commands to reproduce the hang + info on the system you run on.

crossbr commented 5 years ago

I entered that FEN (but without any "Result") and SF_19082608 is up to d=82, 9.5 billion nodes, and no crash yet.

xoto10 commented 5 years ago

Apologies, stall was due to bad memory setting by me :(

vondele commented 5 years ago

right now, I see no issues with this particular fen... will need more info to see if this is an issue / what it is.

xoto10 commented 5 years ago

When it happened at TCEC I would guess it was a hang, there was a msg from cutechess terminating the engine process. It had just under 17 minutes to use, but the longest recent thinks were around 400 and 300 seconds, depths 86/94 and 90/106, 33bn and 22bn nodes.

vondele commented 5 years ago

well, recent multi-threaded tests are all crash free, for 100000s of games. So if it is a real issue, will be hard to find.

xoto10 commented 5 years ago

Sure. Let's wait to see what TCEC say ...

MJZ1977 commented 5 years ago

Perhaps an issue linked to low memory available in comparison with high multithread needs ?

Edit : specially breadcrumbs ? don't know how many memory they need

vondele commented 5 years ago

breadcrumbs use 16K memory, independent of the number of threads ... generally threads don't need much memory (mostly the history tables). It is also no reason to hang.

MichaelB7 commented 5 years ago

Just curious, how did TCEC know it was hung as the result appears to be adjudicated on TCEC and did the TCEC stop play before the time on the clock was used up. If it was hung, it would have lost by time forfeit eventually ( I would think), and there would have been no need to adjudicated the result. So I am curious as to why the result was adjudicated, and if they adjudicated the result before time expired?

xoto10 commented 5 years ago

It ran out of time.

Edit: and was then unresponsive to cutechess so cutechess killed it.

MichaelB7 commented 5 years ago

That was unfortunate as it was a dead draw at that point.

No issue here on my machine ( slightly modified)

dep score   nodes   time    (not shown:  tbhits knps    seldep)
100  -0.26  21.7G   11:25.26    Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Bc6 Kh6 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kg3 Be1+ Kg2 Bc3 Bb5 Bd2 Kf3 Bc3 Bd7 Kf8 Ke2 Kg7 Bc8 Kh6 Kf3 Kh5 Kg3 Be1+ Kh3 Kh6 Ba6 Ba5 Kg4 Kg7 Kg3 Be1+ Kf3 Bc3 B 
 99  -0.26  19.6G   10:16.29    Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Bc6 Kh6 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kg3 Be1+ Kg2 Bc3 Bb5 Bd2 Kf3 Bc3 Bd7 Kf8 Ke2 Kg7 Bc8 Kh6 Kf3 Kh5 Kg3 Be1+ Kh3 Kh6 Ba6 Ba5 Kg4 Kg7 Kg3 Be1+ Kf3 Bc3 B 
 98  -0.26  19.5G   10:11.33    Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Bc6 Kh6 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kg3 Be1+ Kg2 Bc3 Bb5 Bd2 Kf3 Bc3 Bd7 Kf8 Bc8 Bd2 Kg4 Ke7 Kh5 Kf8 Bd7 Kg7 Bc6 Bc3 Be8 Be1 Kg4 Bd2 Kh3 Be1 Bd7 Kh6 Be6 
 97  -0.26  18.8G   9:50.48 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Bc6 Kh6 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kg3 Be1+ Kg2 Bc3 Bb5 Bd2 Kf3 Bc3 Bd7 Kf8 Bc8 Bd2 Kg4 Ke7 Kh5 Kf8 Bd7 Kg7 Bc6 Bc3 Be8 Be1 Kg4 Bd2 Kh3 Be1 Bd7 Kh6 Be6  
 96  -0.26  17.9G   9:23.04 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Bc6 Kh6 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kg3 Be1+ Kg2 Bc3 Bb5 Bd2 Kf3 Bc3 Bd7 Kf8 Bc8 Bd2 Kg4 Ke7 Kh5 Kf8 Bd7 Kg7 Be8 Kf8 Bc6 Kg8 Bb7 Be1 Bc8 Kf8 Bd7 Ke7 Bb5  
 95  -0.26  16.7G   8:44.15 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Ke3 Be1 Kf3 Kg7 g4 hxg4+ Kxg4 Bc3 Kh3 Be1 Bd7 Kh6 Be8 Kg7 Kg2 Kf8 Bb5 Kg7 Bd7 Bd2 Bc8 Be1 Bb7 Ba5 Kf3 Bd2 Kg4 Kh6 Bc8 B 
 94  -0.26  8.93G   4:34.86 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kf3 Bc3 Kf4 Be1 Kg4 Bd2 Kh5 Kf8 Bb5 Ke7 a5 Bxa5 Kh6 Be1 h5 Kf8 Kh7 Bd 
 93  -0.26  6.47G   3:17.75 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kf3 Bc3 Kf4 Be1 Kg4 Bd2 Kh5 Kf8 Bb5 Be1 Bd7 Kg7 Bc6 Kf8 Bb7 Kg7 Kg4 B 
 92  -0.26  5.96G   3:02.11 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Be8 Bd2+ Kh5 Kg7 Kg4 Bc3 Kg3 Be1+ Kh3 Ba5 Bd7 Bd2 Kg4 
 91  -0.26  5.05G   2:34.12 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kf3 Kf8 Bd7 Kg7 Kf4 Bd2+ Kg4 Kf8 Kf3 Be1 Kg2 Kg7 Bb5 Bc3 Kf2 Bd2 Ke2  
 90  -0.26  3.83G   1:56.73 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Bb5 Bd2 Kg4 Ke7 Kg3 Be1+ Kh3 Kd8 Bc6 Ke7 Kg4 Bc3 Kf3 Be1 Ke2  
 89  -0.26  3.69G   1:52.41 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bb4 Kh5 Kg7 Kg4 Bc3 Be8 Bd2 Bc6 Be1 Kf3 Bc3 Ke4 B 
 88  -0.26  3.33G   1:41.81 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Ba5 Kh5 Kg7 Bb5 Kh7 Kg4 Kg7 Kf3 Be1 Ke2 Bb4 Kf2 K 
 87  -0.26  3.15G   1:36.58 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Bc6  
 86  -0.26  2.85G   1:27.58 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Ke2  
 85  -0.26  2.81G   1:26.25 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Ke2  
 84  -0.26  2.79G   1:25.71 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Ke2  
 83  -0.26  2.79G   1:25.57 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Ke2  
 82  -0.26  2.78G   1:25.34 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bd7 Ke7 Bb5 Kf8 Kf3 Ke7 g4 hxg4+ Kxg4 Bc3 Kf4 Be1 Bc6 Kf8 Kg4 Ke7 Kh5 Kf8 Kh6 Kg8 Bd7 Bd2+ Kh5 Kg7 Bc8 Kf8 Kg4 Ke7 Kf3 Kd8 Bb7 Ke7 Ke2  
 81  -0.26  2.77G   1:25.18 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Bd7 Bc3 Kh5 Kf8 Bc6 Kg7 Bb5 Bd2 Ba6 Be1 Bc8 Bd2 Bb7 Bb4 Ba8 Bd2 Kg4 Kh6 Bb7 Kg7 Kf3 Ba5 Kg3 Be1+ Kg2  
 80  -0.26  1.17G   0:36.15 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Bd7 Bc3 Bc6 Ba5 Kg3 Be1+ Kf3 Ba5 Bb5 Be1 Bd7 Bd2 Ke4 Ba5 Kf4 Be1 Be6 Bb4 Kg4 Be1 Bc8 Bd2 Bb7 Bc3 Kf3  
 79  -0.26  1.08G   0:33.38 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Bd7 Bc3 Bc6 Ba5 Kh3 Bc3 Bb5 Ba5 Kg3 Be1+ Kf3 Bc3 Ke4 Bd2 Kd3 Be1 Ke2 Bc3 Kf3 Be1 Ba6 Ba5 Kg4 Be1 Bc8  
 78  -0.26  1.02G   0:31.62 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Bd7 Bc3 Bc6 Ba5 Kh3 Bc3 Bb5 Ba5 Bd7 Bd2 Be8 Be1 Kg2 Ba5 Kf2 Bc3 Ke3 Be1 Kd3 Ba5 Kc2 Kf8 Bc6 Be1 Bd7 B 
 77  -0.26  994.9M  0:30.78 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Bc3 Bd7 Bd2 Kh5 Be1 Be6 Bd2 Bc8 Be1 Bb7 Bc3 Kg4 Be1 Kf3 Bd2 Bc6 Bc3 Ke3 Be1 Ke2 Ba5 Be8 Bc3 Bb5  
 76  -0.26  725.4M  0:22.10 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kf3 Be1 Ke3 Ba5 Bd7 Bc3 Bc6 Be1 Kd3 Kh6 Ke2 Bc3 Kf3 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 Kg7 Kg4 Bc3 Kh5 Bb4 Bb5 
 75  -0.26  687.3M  0:20.81 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kf3 Be1 Ke3 Ba5 Bd7 Bc3 Bc6 Be1 Be8 Bc3 Ke2 Bb4 Bb5 Ba5 Ke3 Be1 Kd3 Ba5 Be8 Be1 Bd7 Ba5 Be6 Kf8 Ke2  
 74  -0.26  654.8M  0:19.77 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kf3 Be1 Ke3 Ba5 Bd7 Bc3 Bc6 Be1 Be8 Bc3 Ke2 Bb4 Bb5 Ba5 Ke3 Be1 Kd3 Ba5 Be8 Be1 Bd7 Ba5 Be6 Kf8 Ke2  
 73  -0.26  645.2M  0:19.46 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Be1 Kf4 Bd2+ Ke4 Be1 Ke3 Bc3 Ke2 Bb4 Bb5 Ba5 Ke3 Be1 Kd3 Ba5 Ke2 Bb4 Kf3 Bc3 Bc6 Bb4 Kg4 Be1 Be8 
 72  -0.26  617.1M  0:18.59 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kf3 Kf8 Ke2 Ba5 Ke3 Kg7 Bd7 Kh6 Kf2 Kg7 Bc6 Bd2 Bb7 Kf8 Kf3 Bc3 Bc6 Kg7 Kg4 Be1 Bb7 Bb4 Bc8 Be1 Kh5  
 71  -0.26  396.3M  0:11.38 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kh5 Ba5 Bc6 Bb4 Bb7 Bd2 Kg4 Be1 Kf3 Bb4 Ba6 Be1 Bc8 Ba5 Ke4 Bd2 Kd3 Be1 Ke2 Bc3 Bd7 Ba5 Bb5 Kf8 Ke3  
 70  -0.26  353.8M  0:10.12 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Be8 Be1 Kd3 Kf8 Ke2 Bb4 Bb5 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Kh5 Ba5 Bc6 Bb4 Bb7 Bd2 Kg4 Be1 Kf3 Bb4 Ba6 Be1 Bc8 Kh6 Ke2 Bb4 Kf2 Bd2 Bd7 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 
 69  -0.26  345.1M  0:09.88 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke3 Be1 Bd7 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Bc3 Kg3 Be1+ Kf3 Ba5 Bc6 Be1 Ke3 Kf8 Kd3 Kg7 Be8 Ba5 Ke4 Be1 Ke3 Kf8 Bc6 Ke7 Kf3 Ba5 Ke4 Bc3 Kf4 Ba5 Kf3 Be1 Kg4 Bd2 Kh3 Be1 Bb7 
 68  -0.26  263.7M  0:07.53 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Be8 Be1 Ke2 Bc3 Kf3 Bb4 g4 Kf8 Bb5 hxg4+ Kxg4 Kg7 Kf3 Bd2 Kg3 Be1+ Kh3 Ba5 Ba6 Be1 Kg4 Bd2 Kh5 Ba5 Bc8 Bc3 Kg4 Kh6 Bb7 Ba5 Kg3 Kh5 Kh3 Kh6 Bc8 Bd2 Bd7 Ba5 Kg4 Be1 Be8 
 67  -0.26  216.3M  0:06.12 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Be8 Be1 Ke2 Bc3 Kf3 Bb4 g4 Kf8 Bb5 hxg4+ Kxg4 Kg7 Kf3 Bd2 Ke2 Ba5 Bc6 Kh6 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Bd7 Kg7 Kg4 Bc3 Kf3 Bd2 Ke2 Ba5 Ke3 Kh6 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Bc 
 66  -0.26  212.0M  0:05.99 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Be8 Be1 Ke2 Bc3 Kf3 Bb4 Bc6 Bd2 g4 hxg4+ Kxg4 Kh6 Bd7 Ba5 Kf4 Kh5 Kg3 Be1+ Kh3 Kh6 Bb5 Kg7 Kg4 Bd2 Kf3 Be1 Ke2 Bc3 Bd7 Ba5 Bc6 Kh6 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Bd7 Bd2 Be 
 65  -0.26  205.2M  0:05.79 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Be8 Be1 Ke2 Bc3 Kf3 Bb4 Bc6 Bd2 g4 hxg4+ Kxg4 Kh6 Bd7 Ba5 Be8 Kg7 Kf4 Bb4 Kf3 Bd2 Bb5 Kh6 Ke2 Bb4 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Kg4 Ba5 Bd7 Bd2 Kg3 Be1+ Kg2 Kh5 Kh3 Kh6 Bb 
 64  -0.26  144.0M  0:04.07 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Bc6 Be1 Kf3 Kh6 g4 Ba5 Bb5 hxg4+ Kxg4 Be1 Bd7 Ba5 Be8 Kg7 Kf4 Bb4 Kf3 Bd2 Bd7 Be1 Kg4 Bc3 Bc6 Kh6 Kf4 Bd2+ Kf3 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 Kg7 Bb5 Bc3 Kg3 Be1+ K 
 63  -0.26  117.5M  0:03.35 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Ke4 Ba5 Bc6 Be1 Kf3 Kh6 g4 Ba5 Bb5 hxg4+ Kxg4 Be1 Bd7 Ba5 Be8 Kg7 Kf4 Bc3 Ke4 Bb4 Bc6 Ba5 Kf3 Be1 Kg4 Kf8 Kh3 Kg7 Be8 Ba5 Bb5 Bd2 Kg4 Kf8 Kf3 Be1 Ke2 Bc3 Kd3 Be1 Bc6  
 62  -0.26  103.2M  0:02.96 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf2 Kf8 Bb5 Bc3 Bd7 Bd2 Ke2 Bc3 Kf3 Kg7 g4 hxg4+ Kxg4 Kh6 Kf3 Kh5 Kg3 Be1+ Kh3 
 61  -0.26  88.5M   0:02.53 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf3 Kf8 g4 hxg4+ Kxg4 Kg7 Kf4 Bc3 Ke4 Kh6 Kf3 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 Kg7 Bd 
 60  -0.26  80.7M   0:02.31 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf3 Kf8 Bb5 Be1 g4 hxg4+ Kxg4 Bd2 Kf3 Kg7 Bc6 Bc3 Kg4 Bd2 Kg3 Be1+ Kh3 Bd2 Kg 
 59  -0.26  67.6M   0:01.93 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf3 Kf8 Bb5 Kg7 g4 hxg4+ Kxg4 Bd2 Be8 Be1 Kh5 Bd2 Bc6 Kf8 Kg4 Be1 Kf3 Bc3 Bd7 
 58  -0.26  63.6M   0:01.81 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf3 Kf8 Bb5 Kg7 g4 hxg4+ Kxg4 Bd2 Kf3 Be1 Ke3 Bc3 Bc6 Kh6 Kf3 Kg7 Kg4 Ba5 Kf4 
 57  -0.26  55.7M   0:01.56 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Ba5 Kf3 Kf8 Bb5 Be1 g4 hxg4+ Kxg4 Bb4 Kf3 Be1 Bd7 Bb4 Ke2 Bc3 Kd3 Be1 Bc6 Bb4 Bb5 
 56  -0.26  48.0M   0:01.33 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Bb4 Kf3 Kf8 Bb5 Be1 g4 hxg4+ Kxg4 Bb4 Bc6 Kg7 Kf3 Be1 Bb5 Kf8 Ke2 Ba5 Ke3 Kg7 Ke4 
 55  -0.26  45.9M   0:01.27 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Bb4 Kf3 Kf8 Bb5 Be1 g4 hxg4+ Kxg4 Bb4 Bc6 Ba5 Kg3 Be1+ Kf3 Kg7 Kg4 Bb4 Be8 Kf8 Bb 
 54  -0.26  45.3M   0:01.25 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Ke7 Ke2 Bb4 Kf3 Kf8 Bb5 Be1 g4 hxg4+ Kxg4 Kg7 Be8 Bb4 Kf3 Kf8 Bb5 Kg7 Kf2 Bd2 Bd7 Bc3 Bc6 
 53  -0.26  41.4M   0:01.14 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Kc2 Be1 Kd3 Kg7 Bd7 Ba5 Ke4 Kf8 g4 hxg4 Kf4 Be1 Kxg4 Kg7 Be8 Ba5 Kf4 Kg8 Bc6 Kg7 Kf3 Be1 Ke2 
 52  -0.26  31.5M   0:00.86 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Bc6 Bb4 Bb7 Kg7 Ke2 Bc3 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Be1 Bc6 Kh6 Bb5 Ba5 Kf3 Bc3 Be8 Kg7 Kf4 
 51  -0.26  24.7M   0:00.66 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Kc2 Ke7 Bc6 Bb4 Kd3 Kf8 Ke4 Kg7 g4 hxg4 Kf4 Bd2+ Kxg4 Ba5 Kf4 Bd2+ Ke4 Kh6 Kf3 Be1 Ke2 Ba5 Kf2 Kh5 Kg3 Be1+ K 
 50  -0.26  21.6M   0:00.58 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Kc2 Ke7 Bc6 Bb4 Kd3 Ba5 Ke2 Bb4 Kf3 Kf8 g4 hxg4+ Kxg4 Bd2 Kf3 Be1 Ke2 Ba5 Kd3 Be1 Bb5 Ke7 h5 Kf8 Bc6 Kg7 
 49  -0.26  18.8M   0:00.51 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kc1 Bc3 Be8 Kf8 Kc2 Be1 Kd1 Bb4 Bb5 Ba5 Kc2 Ke7 Bc6 Bb4 Kd3 Ba5 Ke2 Kf8 g4 hxg4 Kf2 Bd2 Kg3 Kg7 Kxg4 Bc3 Bb5 Bd2 Kf3 Kh6 Be8 Kg7 
 48  -0.26  15.8M   0:00.43 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Ke4 Kf8 Bc6 Ba5 Kf4 Bd2+ Kf3 Kg7 g4 hxg4+ Kxg4 Bc3 Bb5 Be1 Be8 Bc3 Kf3 Kf8 Bd7 Bd2 Ke2 Bc3 Kf2 Bb4 Kg3 Be1+ Kg4 Kg7 Bb5 Bc3 
 47  -0.26  13.9M   0:00.38 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kd1 Ba5 Bb5 Bc3 Ke2 Ba5 Be8 Kf8 Bd7 Kg7 Kf3 Bb4 g4 hxg4+ Kxg4 Ba5 Kf3 Kf8 Bb5 Bb4 Kg4 Ba5 Kh3 Bd2 Kg3 Be1+ Kf4 Kg7 Bd7 Bb4 Kg3 
 46  -0.26  11.8M   0:00.32 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kd1 Ba5 Bb5 Bc3 Ke2 Ba5 Be8 Bb4 Kf2 Bc3 Kf3 Bb4 g4 Kf8 Bb5 hxg4+ Kxg4 Bd2 Kf3 Ba5 Ke2 Kg7 Bd7 Kh6 Be8 Kh5 Kf3 Kxh4 
 45  -0.26  10.5M   0:00.29 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Kd1 Ba5 Bb5 Bc3 Ke2 Ba5 Be8 Bb4 Kf2 Ba5 Bc6 Bd2 Ke2 Ba5 Kf3 Kh6 Bd7 Kg7 g4 hxg4+ Kxg4 Bb4 Kf4 Bd2+ Kg3 Ba5 Kf3 Kh6 Bc6 
 44  -0.26  9.27M   0:00.25 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Bb4 Ke4 Kf8 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Bc6 Be1 Be8 Bc3 Kf3 Ba5 Kf2 Bd2 Ke2 Ba5 Bc6 Kh6 
 43  -0.26  7.59M   0:00.21 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Bb4 Ke4 Kf8 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Bc6 Be1 Kf3 Kh6 Ke2 Ba5 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 
 42  -0.26  7.12M   0:00.20 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Bb4 Ke4 Kf8 Kf3 Ba5 g4 hxg4+ Kxg4 Kg7 Kf4 Bd2+ Kf3 Bb4 Bb5 Bd2 Ke2 Bb4 Kf2 Ba5 Be8 
 41  -0.26  6.06M   0:00.16 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Bb4 Ke4 Bd2 Bc6 Be1 Kf3 Kh6 Bd7 Bc3 g4 hxg4+ Kxg4 Kg7 Kf3 Kf8 Bb5 Bb4 Bc6 Kg7 Ke2 
 40  -0.26  5.44M   0:00.15 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Ba5 Bc6 Bb4 Ke2 Bc3 Kf2 Bb4 Kf3 Kh6 g4 hxg4+ Kxg4 Bd2 Bb5 Ba5 Be8 Kg7 Kf3 
 39  -0.26  4.89M   0:00.13 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Ba5 Bc6 Bb4 Ke4 Be1 Kf3 Kh6 Bd7 Kg7 g4 hxg4+ Kxg4 Bd2 Be8 Be1 Kf3 Kf8 
 38  -0.26  4.29M   0:00.12 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Kc2 Be1 Bb5 Bb4 Kd3 Be1 Bd7 Ba5 Bc6 Bb4 Ke4 Be1 Kf3 Bd2 g4 hxg4+ Kxg4 Be1 Bd7 Ba5 Kf3 Bd2 Ke2 
 37  -0.26  3.91M   0:00.11 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Bb4 Kf3 Ba5 Be8 Bd2 Ke2 Bb4 Bc6 Bc3 Kf2 Kh6 Be8 Kg7 Bd7 Kf8 Kf3 
 36  -0.26  3.28M   0:00.09 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Bc3 Kf3 Bb4 Ke2 Kh6 Kf2 Kh5 Kg3 Be1+ Kh3 Kh6 Be8 Kg7 Bb5 Ba5 Kg2 Bd2 
 35  -0.26  2.94M   0:00.08 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Bc3 Kf3 Be1 Be8 Bd2 Kf2 Ba5 Ke2 Bb4 Kd1 Kf8 Bd7 Kg7 Kc2 
 34  -0.26  2.62M   0:00.07 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Be1 Be8 Bd2 Bb5 Bc3 Kg3 Be1+ Kf3 Ba5 Bc6 Bd2 Ke2 
 33  -0.26  2.37M   0:00.07 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Be1 Be8 Bc3 Bd7 Bd2 Kf3 Ba5 Bc6 Bb4 Bb5 Bd2 
 32  -0.26  2.17M   0:00.06 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Bb4 Bd7 Bd2 Kf3 Bc3 Bb5 Ba5 Bc6 Bd2 
 31  -0.26  2.02M   0:00.06 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Bb4 Bd7 Kf8 Bb5 Kg7 Kf4 Bd2+ Kf3 Ba5 Bc6 
 30  -0.26  1.88M   0:00.05 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Bd2 Be8 Bb4 Kf4 Bd2+ Kf3 Ba5 Ke4 Kf8 
 29  -0.26  1.74M   0:00.05 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Ba5 Bb5 Be1 Be8 Ba5 Kf3 
 28  -0.26  1.66M   0:00.05 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Ba5 Bb5 Be1 Be8 Bd2 Bd7 Be1 Kf3 Bd2 
 27  -0.26  1.60M   0:00.05 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Bb4 Bb5 Ba5 Be8 Be1 Kf4 
 26  -0.26  1.52M   0:00.04 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Ke4 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Kf8 Bb5 Kg7 Be8 Bd2 Kf3 
 25  -0.26  1.42M   0:00.04 Kf8 Kf2 Bb4 Kf3 Bd2 Bb5 Kg7 Ke2 Bc3 Kd1 Ba5 Kc2 Be1 Kd3 Bb4 Bc6 Ba5 Be8 Kf8 Bb5 Bb4 Ke2 Bc3 Bc6 Bb4 Kf2 Kg7 Kf3 Bc3 Be8 Ba5 Bd7 Bc3 g4 hxg4+ Kxg4 Bd2 
 24  -0.27  1.24M   0:00.04 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Be1 Kf3 Kg7 Be8 Ba5 Kf4 Bd2+ Kg3 Ba5 Bb5 Bd2 Kf3 Be1 h5 Ba5 Be8 Bd2 Kg3 Kf8 Bb5 
 23  -0.27  1.00M   0:00.03 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Be1 Kf3 Ba5 Ke4 Kf8 Bc6 Kg7 Bb5 Kh6 Kf4 Bd2+ Kf3 Kg7 Ke2 Ba5 Kf2 Bc3 
 22  -0.27  926537  0:00.03 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Be1 Kf3 Ba5 Ke4 Kf8 Bb5 Kg7 Kf4 Be1 Kf3 Bd2 Ke2 Ba5 Bc6 Kh6 Bd7 Kh5 
 21  -0.27  897020  0:00.03 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Bd7 Be1 Kf3 Bd2 Bb5 Be1 Bc6 Kh6 Kg4 Kg7 Bd7 Kf8 Kf4 Bd2+ Kg3 Kg7 Kf3 Be1 
 20  -0.27  867981  0:00.03 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Be1 Kf3 Ba5 Ke4 Kf8 Bb5 Kg7 Kf4 Be1 Kf3 Bd2 
 19  -0.28  846102  0:00.03 Kf8 Kf2 Bd2 Kf3 Kg7 g4 hxg4+ Kxg4 Be1 Be8 Ba5 Kf4 Be1 Ke4 Ba5 h5 Bd2 Kf3 Be1 Bc6 Ba5 
 18  -0.31  610642  0:00.02 Kf8 Kf2 Bd2 Kf3 Be1 g4 hxg4+ Kxg4 Kg7 Bd7 Kf8 Kf3 Ba5 Bb5 Be1 Bc6 Kg7 Be8 Ba5 Kf4 Be1 Ke4 
 17  -0.31  586951  0:00.02 Kf8 Kf2 Bb4 Bb5 Kg7 Kf3 Ba5 Be8 Kf8 Bc6 Be1 g4 hxg4+ Kxg4 Kg7 Bd7 Bd2 Be8 Ba5 Kf4 Be1 Kf3 Bc3 
 16  -0.27  378013  0:00.01 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Kf8 Bb5 Kg7 Bc6 Ba5 Kf4 Kh6 Be8 Kh5 Bxf7+ Kxh4 
 15  -0.30  315089  0:00.01 Kf8 Kf2 Bb4 Kf3 Bd2 g4 hxg4+ Kxg4 Kg7 Be8 Kf8 Bb5 Be1 Kf4 Kg7 Bc6 Kh6 Kg4 Kg7 Be8 
 14  -0.31  264021  0:00.01 Kf8 Kf1 Kg7 Kg2 Bb4 Kf3 Be1 g4 hxg4+ Kxg4 Bd2 h5 Be1 Be8 Bd2 Kg3 Kg8 Bb5 Kg7 
 13  -0.31  241285  0:00.01 Kf8 Kg2 Bd2 Kf3 Be1 Bd7 Kg7 Be8 Kf8 Bc6 Kg7 Bb7 Bd2 g4 hxg4+ Kxg4 Bc3 
 12  -0.32  171209  0:00.01 Kf8 Kf2 Kg7 Kg2 Bc3 Kf3 Be1 g4 hxg4+ Kxg4 Bd2 h5 Bc3 Kf4 
 11  -0.33  140075  0:00.01 Kf8 Kf2 Kg7 Kf3 Bd2 g4 hxg4+ Kxg4 Kh6 Be8 Kg7 h5 
 10  -0.36  88917   0:00.00 Kf8 Kf1 Kg7 Kf2 Bb4 Kf3 Bc3 g4 hxg4+ Kxg4 Kh6 Bd7 
  9  -0.32  66143   0:00.00 Kf8 Kf1 Kg7 Ke2 Bc3 Kf3 Bd2 g4 hxg4+ Kxg4 Kh6 
  8  -0.44  52361   0:00.00 Kf8 Kf2 Kg8 Kf3 Bd2 g4 hxg4+ Kg3 Kh7 
  7  -0.42  30033   0:00.00 Kf8 g4 hxg4 Kf2 g3+ Kxg3 Kg7 
  6  -0.38  14349   0:00.00 Kd8 Kf2 Kc7 Be8 Bc3 Bxf7 
  5  -0.31  7651        0:00.00 Be1 Kg2 Bd2 Kf3 Be1 
  4  -0.27  6157        0:00.00 Kd8 Kf2 Kc7 
  3  -0.37  4237        0:00.00 Be1 Kh2 Bd2 
  2  -0.20  2508        0:00.00 Kd8 Kg2 
  1  -0.08  951         0:00.00 Be1 
  0 #
xoto10 commented 5 years ago

In the absence of other evidence TCEC have ruled it as a stockfish crash (well, hang really).

A viewer said this was the last output, posted after about 22s of thought:

<Stockfish 190826(13): info depth 83 seldepth 114 multipv 1 score cp -41 nodes 2661493724 nps 121020995 hashfull 97 tbhits 1495869 time 21992 pv e7f8 g1f2 a5c7 f2f3 f8e7 c6b7 c7a5 g3g4 h5g4 f3g3 a5c3 g3g4 c3a5 b7a6 a5c3 a6b5 c3d2 b5c6 d2e1 c6a8 e1a5 g4h5 a5d2 a8c6 d2e1 h5g4 e1d2 c6b5 d2a5 g4h3 a5d2 b5a6 d2c3 a6c8 c3e1 c8b7 e1d2 h3g2 d2c3 b7c6 c3d2 g2f1 d2c3 c6a8 c3d2 f1f2 e7f8 f2e2 d2b4 e2f3 f8e7 f3f2 b4d2 a8b7 e7f8 f2f3 d2e1 b7c6 f8e7 f3e2 e1b4 c6b5 ............

Roughly 16 mins later, after running out of time:

Stockfish 190826(13): stop Terminating process of engine Stockfish 190826(13)

So it sounds like something happened while it was in its normal window of thinking time that made it not send any more output to cutechess. That could have been when it tried to send its move, of course, or it could be some kind of internal infinite loop, or ...

In fact previous moves took 23/22/36/22/21 seconds, so 22s sounds like when it tried to send a move back to cutechess?

MJZ1977 commented 5 years ago

In fact previous moves took 23/22/36/22/21 seconds, so 22s sounds like when it tried to send a move back to cutechess?

This explanation seems the most logic. Specially the position has no reason to lead to infinite loops. But is it a SF issue or cutechess issue. I don't know ... Perhaps SF sended the move but cutechess didn't receive it.

Aloril commented 5 years ago

Whole log for this game is at https://tcec-chess.com/crash/s16divp_51.7z (= http://tinyurl.com/y597lyyp listed in Crash info)

snicolet commented 5 years ago

Specially the position has no reason to lead to infinite loops.

Maybe not infinite loops, but search explosion?

The last output received from Stockfish was at depth 82 and selective depth 114 after (almost) 22 seconds, and then nothing happened in like 15 minutes before Stockfish was flagged:

82128711 <Stockfish 190826(13): info depth 83 seldepth 114 multipv 1 score cp -41 nodes 2661493724 nps 121020995 hashfull 97 tbhits 1495869 time 21992 pv e7f8 g1f2 a5c7 f2f3 f8e7 c6b7 c7a5 g3g4 h5g4 f3g3 a5c3 g3g4 c3a5 b7a6 a5c3 a6b5 c3d2 b5c6 d2e1 c6a8 e1a5 g4h5 a5d2 a8c6 d2e1 h5g4 e1d2 c6b5 d2a5 g4h3 a5d2 b5a6 d2c3 a6c8 c3e1 c8b7 e1d2 h3g2 d2c3 b7c6 c3d2 g2f1 d2c3 c6a8 c3d2 f1f2 e7f8 f2e2 d2b4 e2f3 f8e7 f3f2 b4d2 a8b7 e7f8 f2f3 d2e1 b7c6 f8e7 f3e2 e1b4 c6b5 b4a5 b5a6 a5c3 a6c8 c3b4 e2f2 e7f8 f2f3 b4c3 c8e6 f8g7 f3g4 c3d2 g4g3 d2c3 e6c8 c3e1 g3f3 g7h6 f3g4 e1a5 c8b7 a5e1 b7c6 h6g7 c6b5 e1a5 g4f4 g7f8 b5d7 a5d2 f4f3 d2a5 h4h5 f8g7 h5h6 g7h6

But already at depth 29 we were reaching selective depth 113:

82106994 <Stockfish 190826(13): info depth 29 seldepth 113 multipv 1 score cp -41 nodes 852215 nps 85221500 tbhits 0 time 10 pv e7f8 g1f2 a5c7 f2f3 f8e7 c6b7 c7a5 g3g4 h5g4 f3g3 a5c3 g3g4 c3e1 g4h5 e1b4 h5h6 e7f8 b7c8 b4e1 c8d7 e1d2 h6h5 d2e1 d7c6 e1d2 c6b7 f8e7 b7a6 d2b4 a6b5 b4e1 h5g4 e1a5 b5c6 a5d2 g4f3 d2e1 f3e2 e1b4 e2d1 b4a5 c6b7 a5c3 b7c8 e7f8 d1e2 f8e7 c8a6 c3b4 h4h5 b4a5 e2f2

Maybe I am wrong, but the sort of position when this happened (shuffling in a draw opposite colored bishops endgame with score = 0.41, stockfish having not yet shown 0.00) seems to hint that maybe we observed a search explosion problem somewhere, maybe related to the shuffling patch?

I wasn't really active when the shuffling patch was developed, but did we had problems like that during its development or tuning?

Mindbreaker1 commented 5 years ago

It went fine through depth 85. Then I ran it again and I got some sort of memory leak at depth 79. Speed went down to a crawl. Third time seems to be just fine again. I am using Fritz 16 interface.

vondele commented 5 years ago

unlikely that shuffle extensions are related. Threads will quit search if they observe the signal stop. I.e. pass at line 1199 in search. Similarly, stop will be signaled by the mainthread if that one reaches line 1828. It isn't very easy to prove this happens always, but maybe there is some very rare case (that I can't see, but why is the cycle detection code before the time check)?

Memory corruption could of course cause anything, e.g. if some random write causes ponder to be true, this would happen.

vondele commented 5 years ago

@Mindbreaker1 with or without TB ?

Mindbreaker1 commented 5 years ago

With 6-man and less

Mindbreaker1 commented 5 years ago

I had the hash set at 12+ GB Tried a few more times. Haven't got it to do it again.

snicolet commented 5 years ago

Threads will quit search if they observe the signal stop

Hmmm, maybe our code is not robust enough for time emergency, by the way. It is true that we exit the search in line 1199 in any thread if the main thread has signaled in check_time(), but line 1199 is after closing a subtree, so there could still be an (infinite) chain of subtree openings before we stop.

I would prefer to add these lines around line 1199 when entering the search function, to be honest:

if (Threads.stop.load(std::memory_order_relaxed))
    return VALUE_ZERO;

Of course that would just be to be 100% sure that our time emergency procedure is correct in case of explosion problem, but that doesn't fix the underlying problem.

vondele commented 5 years ago

I really think this wasn't a time emergency. max_time (which is checked by check_time) must have been much shorter than the available time on the clock (would need to verify, but I'm rather sure, additionally with have the 1s moveoverhead).

I agree we should be defensive in the code, but before we rush to a solution we should understand what is going on.

Mindbreaker1 commented 5 years ago

Well, I got another unusual result. It stayed on 79 much longer, but everything else looked fine.

snicolet commented 5 years ago

@vondele What makes me really suspiscious is the depth 29 / selective depth 119 reported in the logs

xoto10 commented 5 years ago

I had a 10 minute gap in writes to the log while testing (11 slow cores, but still):

Note the ^C because nothing seemed to be happening and the date at that point compared to output of ls:

$ tail -f sfmaster.log << info depth 87 currmove a5d2 currmovenumber 2 << info depth 87 currmove a5b4 currmovenumber 3 << info depth 87 currmove a5c3 currmovenumber 4 << info depth 87 currmove a5e1 currmovenumber 5 << info depth 87 currmove e7d8 currmovenumber 6 << info depth 87 currmove a5d8 currmovenumber 7 << info depth 87 currmove a5c7 currmovenumber 8 << info depth 87 currmove a5b6 currmovenumber 9 << info depth 87 seldepth 140 multipv 1 score cp -42 nodes 18627970197 nps 17266666 hashfull 1000 tbhits 0 time 1078840 pv e7f8 g1f2 f8g7 c6d7 g7f8 f2e3 f8e7 d7b5 a5c3 e3e2 e7f8 b5a6 f8e7 e2f2 c3a5 f2e3 a5e1 e3f3 e7d8 a6b7 e1a5 f3g2 d8c7 b7c6 c7d8 c6b5 d8e7 g2h3 e7d8 b5a6 d8e7 a6b7 e7d7 h3h2 d7e7 h2g2 a5e1 b7c6 e1a5 g2f3 a5d2 f3e4 e7d8 c6b5 d8e7 b5a6 d2c3 a6b7 c3a5 b7c6 a5e1 e4e3 e1c3 e3d3 c3e1 d3e2 e1c3 c6a8 c3a5 e2f1 e7d8 f1g1 d8e7 g1h1 e7d8 a8b7 d8c7 b7a6 c7d8 h1g2 d8c7 g2f2 c7d7 f2f3 a5e1 a6b5 d7d8 a4a5 e1a5 f3f2 a5b4 b5a4 b4c3 f2g2 d8e7 a4d1 e7d7 d1h5 << info depth 88 currmove e7f8 currmovenumber 1 ^C $ ls -lt total 6814112 -rw------- 1 sf sf 48842 Sep 9 16:19 sfmaster.log -rw------- 1 sf sf 297 Sep 9 16:01 sfnew.log -rw------- 1 sf sf 656 Sep 9 16:01 cute_2h_master.out drwx------ 3 sf sf 24576 Sep 9 16:01 ./ -rwx------ 1 sf sf 1620 Sep 9 16:01 cute_2h* ... $ date Mon 9 Sep 16:29:44 BST 2019

Output continued very shortly after:

$ tail -f sfmaster.log << info depth 87 currmove a5d2 currmovenumber 2 << info depth 87 currmove a5b4 currmovenumber 3 << info depth 87 currmove a5c3 currmovenumber 4 << info depth 87 currmove a5e1 currmovenumber 5 << info depth 87 currmove e7d8 currmovenumber 6 << info depth 87 currmove a5d8 currmovenumber 7 << info depth 87 currmove a5c7 currmovenumber 8 << info depth 87 currmove a5b6 currmovenumber 9 << info depth 87 seldepth 140 multipv 1 score cp -42 nodes 18627970197 nps 17266666 hashfull 1000 tbhits 0 time 1078840 pv e7f8 g1f2 f8g7 c6d7 g7f8 f2e3 f8e7 d7b5 a5c3 e3e2 e7f8 b5a6 f8e7 e2f2 c3a5 f2e3 a5e1 e3f3 e7d8 a6b7 e1a5 f3g2 d8c7 b7c6 c7d8 c6b5 d8e7 g2h3 e7d8 b5a6 d8e7 a6b7 e7d7 h3h2 d7e7 h2g2 a5e1 b7c6 e1a5 g2f3 a5d2 f3e4 e7d8 c6b5 d8e7 b5a6 d2c3 a6b7 c3a5 b7c6 a5e1 e4e3 e1c3 e3d3 c3e1 d3e2 e1c3 c6a8 c3a5 e2f1 e7d8 f1g1 d8e7 g1h1 e7d8 a8b7 d8c7 b7a6 c7d8 h1g2 d8c7 g2f2 c7d7 f2f3 a5e1 a6b5 d7d8 a4a5 e1a5 f3f2 a5b4 b5a4 b4c3 f2g2 d8e7 a4d1 e7d7 d1h5 << info depth 88 currmove e7f8 currmovenumber 1 << info depth 88 currmove a5d2 currmovenumber 2 << info depth 88 currmove a5e1 currmovenumber 3 << info depth 88 currmove a5b4 currmovenumber 4 << info depth 88 currmove a5c3 currmovenumber 5 << info depth 88 currmove e7d8 currmovenumber 6 << info depth 88 currmove a5d8 currmovenumber 7 << info depth 88 currmove a5b6 currmovenumber 8 << info depth 88 currmove a5c7 currmovenumber 9 << info depth 88 seldepth 96 multipv 1 score cp -42 nodes 32787387402 nps 16933010 hashfull 1000 tbhits 0 time 1936300 pv e7f8 g1f2 f8g7 c6d7 g7f8 f2e3 f8e7 d7b5 a5c3 e3e2 e7f8 b5a6 f8e7 e2f2 c3b4 f2g2 b4a5 a6b5 e7f8 g2f3 a5c3 f3e4 c3a5 e4e3 f8e7 e3f3 a5d2 b5c6 d2c3 f3e2 e7f8 e2d3 c3e1 d3e3 f8g7 c6b7 g7h6 e3d3 e1a5 b7c6 h6g7 c6d7 a5e1 d3e4 g7f8 e4e3 f8e7 d7c6 e1c3 e3f3 c3e1 f3e4 e1a5 c6b7 a5e1 e4f4 e1d2 f4f3 d2c3 f3f2 c3a5 b7c6 e7d8 f2e3 d8e7 c6a8 a5e1 e3f4 e1d2 f4f3 d2c3 f3f2 c3a5 a8b7 e7d8 b7a6 d8c7 f2e3 a5e1 g3g4 h5g4 h4h5 g4g3 e3f3 e1d2 f3g3 c7d7 g3f2 d7e7 f2e2 d2b4 << info depth 89 currmove e7f8 currmovenumber 1

Does hashful 1000 indicate a full hash? I was using 8GB (option.Hash=8192)

Alayan-stk-2 commented 5 years ago

Yes hashfull 1000 is 100% utilization of hash. Hashfull 123 is 12.3% full.

xoto10 commented 5 years ago

I put some debug output in and maximumtime looks to be around say 15-20% of total time, so if the check is made, it should stop us in plenty of time. This matches my experience watching TCEC where I would say sf takes a max of about 33% of available time. (Some engines occasionally get up towards 50%)

vondele commented 5 years ago

@vondele What makes me really suspiscious is the depth 29 / selective depth 119 reported in the logs

that's quite usual with the shuffle extensions, and was discussed in depth. This should still not cause the time_check mechanism to fail (if that's what is going on).

Mindbreaker1 commented 5 years ago

OK, the memory leak happened again. But it happened at depth 90.
I doubt the stuff I have pasted below is useful, but what do I know:

Stockfish 030919 64 POPCNT 8/4kp2/2Bp1p2/b1pP1P1p/P1P4P/6P1/8/6K1 b - - 0 1

Analysis by Stockfish 030919 64 POPCNT:

57...Kf8 58.Kg2 Be1 59.Kf3 Kg7 60.Bd7 Bc3 61.Ke2 Kh7 62.Kd3 Be1 63.Bc6 Kg7 64.Bb7 Bb4 65.Bc8 Be1 66.Ke2 Bc3 67.Ba6 Bb4 68.Kf3 Bc3 69.Ke4 Be1 70.Bb7 Kh6 71.Kd3 Kg7 72.Kc2 Kh6 73.Bc6 Kg7 74.Bb5 Kh6 75.Be8 Kg7 76.Kd1 Bc3 77.Ke2 Kf8 78.Bb5 Ke7 79.Bc6 Ba5 80.Ke3 Be1 81.Kf3 Kf8 82.Ke2 Ba5 83.Kd3 Kg7 84.Bb5 Be1 85.Ke2 Bb4 86.Ba6 Bc3 87.Kf3 Ba5 88.Bb7 Bb4 89.Kf2 Bc3 90.Bc6 Bb4 91.Be8 Bc3 92.Bb5 Kf8 93.Kf3 Kg7 94.Bd7 Bb4 95.Be8 Kf8 96.Bb5 Kg7 97.Kg2 Bc3 98.Bc6 Ba5 99.Kh2 Be1 100.Bb5 Bb4 101.Bd7 Kf8 102.Kh3 Kg7 103.Be8 Bc3 104.Bc6 Kh6 105.g4 hxg4+ 106.Kxg4 Kg7 White is slightly better: +/= (0.45) Depth: 89/99 00:22:37 25076MN, tb=68103924 (, 09.09.2019)

Things started to go wonky at about 5 minutes past this output.

vondele commented 5 years ago

@Mindbreaker1 ... if you say memory leak, how do you measure / observe that ?

What is sometimes seen is that the OS caches access to TB files, but that's more of an OS 'feature'. Do you see the same behavior without TB ?

vondele commented 5 years ago

As a test of the timing mechanism, I run the position on a similar setup (~threads, hash, fen, but no TB), and do a go movetime 25000. This uses exactly the same mechanism to quit search as maximumTime. With 10 tests, search finishes within 3ms of the time limit, which is OK:

time 25001 
time 25003 
time 25001 
time 25003 
time 25001 
time 25003 
time 25002 
time 25001 
time 25001 
time 25001

Edit: similar statistics for ~100 runs:

     76 time 25001 
     18 time 25002 
      2 time 25003 
      2 time 25004
Mindbreaker1 commented 5 years ago

The Windows Task Manager showed far less memory in the "details" tab for Stockfish than it was showing earlier and the memory for everything seemed to be reducing. Everything was glitching. I noticed and investigated after the music I was listening to started to sputter. The machine has 16 GB of RAM. Fritz allowed a maximum of 12288 MB and that is what I chose.

This is still with the 6-man tables.

It looks like I allowed the GUI table access too.

I am using 8 of 16 threads.

I have had this Fritz 16 version for less than a week, so it could just be glitches in their program. I did sit on some positions for hours though with no issues.

xoto10 commented 5 years ago

Log not updated for 22+ mins (16GB hash now, still waiting as I type):

<< info depth 93 currmove e7d8 currmovenumber 9 << info depth 93 seldepth 128 multipv 1 score cp -42 nodes 34207263908 nps 16336010 hashfull 1000 tbhits 0 time 2093979 pv e7f8 g1f1 f8g8 f1f2 a5d2 f2e2 d2a5 e2f3 g8f8 f3e4 f8g7 e4d3 g7h6 d3c2 a5b4 c6b5 b4a5 b5e8 h6g7 e8d7 g7f8 c2d1 a5c3 d7c8 f8e8 d1e2 e8f8 e2d3 c3b4 c8b7 b4e1 d3e2 e1c3 e2f3 f8g7 f3e4 c3e1 e4e3 g7f8 b7c6 f8e7 e3e4 e1b4 c6b7 e7d8 e4e3 b4e1 e3f3 e1c3 b7c6 d8e7 f3e2 c3a5 e2f2 e7f8 c6b5 f8g7 b5d7 a5c3 f2f3 g7h6 f3g2 h6g7 d7b5 g7f8 b5a6 c3a5 g2f1 a5c3 f1f2 f8g8 f2e2 g8g7 a6b5 g7h6 e2f2 c3d2 f2f3 h6h7 g3g4 h5g4 f3g4 h7h6 b5c6 d2c3 c6e8 h6g7 g4f3 g7f8 e8b5 f8g8 b5c6 g8h7 c6e8 << info depth 94 currmove e7f8 currmovenumber 1 ^C $ $ lh -2 total 6814120 -rw------- 1 sf sf 52673 Sep 9 20:37 sfmaster.log $ $ date Mon 9 Sep 20:59:45 BST 2019

vondele commented 5 years ago

@xoto10 if this is happening could you see if SF is responsive to issuing a stop ? It should react instantly with a bestmove output.

xoto10 commented 5 years ago

No, I'm running via cutechess :( It is still using cpu, and experience suggests it is still working, it just takes 2 or 3 minutes inbetween iterations sometimes (and then occasioanlly much longer). I have wondered if there is buffering of the output, but this file is named directly to cutechess so I don't think so. Edit: this is using tc=inf cutechess-cli -openings file=crash1_20190909.pgn -resign movecount=3 score=400 -draw movenumber=40 movecount=5 score=5 -games 1 -rounds 1 -pgnout cute_2h_master.pgn -ratinginterval 100 -engine cmd=stockfish_master name=master 'option.Debug Log File=sfnew.log' option.Hash=16 -engine cmd=stockfish_master name=master 'option.Debug Log File=sfmaster.log' option.Hash=16384 -each tc=inf proto=uci option.Threads=11 'option.Move Overhead=50' 'option.Minimum Thinking Time=5' -concurrency 1

I think I set up Arena to use the server remotely once, let me look ...

MJZ1977 commented 5 years ago

@xoto10 : did you set a maximum time + increment ?

@snicolet : I don't think it is related to shuffle extension. In generale these hangs can be reproduced easily for the same position and can be explained. In all cases I have seen, it never exceed maximum time as @vondele explained. I think that SF didn't send the bestmove information for a reason that I can't understand. sync_cout << "bestmove " << UCI::move(bestThread->rootMoves[0].pv[0], rootPos.is_chess960());

Or it is another problem in cutechess or interface with cutechess. I don't know also if different compiled can have a "special" wrong behavior.

xoto10 commented 5 years ago

It's moved on now, started depth 96 at 21:11

vondele commented 5 years ago

@xoto10 .... you should try the command line :-) just paste something like:

uci
setoption name Hash value 16384
setoption name Move Overhead value 1000
setoption name Threads value 31
isready
ucinewgame
setoption name Ponder value false
position startpos moves c2c4 e7e5 g2g3 g8f6 b1c3 f8c5 f1g2 b8c6 e2e3 e8g8 g1e2 d7d6 d2d4 c5b6 e1g1 c8f5 h2h3 f8e8 a2a3 a7a5 b2b3 h7h6 a1a2 f5h7 a2d2 d8d7 c1b2 b6a7 f1e1 g8h8 d1c1 h8g8 c3d5 f6e4 d2d1 e5d4 e3d4 c6e7 e2c3 e7d5 c3e4 h7e4 g2e4 d5f6 e4b7 a8b8 e1e8 f6e8 b7g2 b8b3 c1c2 a5a4 d1d3 b3b8 h3h4 d7f5 d3d2 f5c2 d2c2 c7c5 d4d5 a7b6 g2h3 e8f6 b2f6 g7f6 c2b2 b6c7 b2b8 c7b8 h3d7 g8g7 f2f4 b8c7 f4f5 c7a5 d7a4 a5c3 a4c6 g7f8 g1g2 f8g7 c6b5 g7f8 g2f3 f8g7 a3a4 c3d2 b5e8 d2e1 f3e2 e1c3 e8d7 h6h5 d7e6 c3a5 e2f3 a5d2 e6d7 d2e1 d7e8 e1c3 e8c6 c3a5 f3e4 a5e1 e4e3 g7f8 e3e2 e1a5 e2f2 f8e7 f2g1
isready
go movetime 25000

however, I think the long time between output is reasonable at depth 93... roughly the same time as the time used so far would be OK IMO (so about 2093979ms in your example).

xoto10 commented 5 years ago

Seems to respond to stop - came back with the pv for the next depth and all cpu activity stopped. Will look at it some more tomorrow ...

Yeah, I did think of running sf directly, wasn't sure I would get the list of commands right :)

snicolet commented 5 years ago

Looking at the so-called "Livelog" tab of TCEC during the game Stockfish-KomodoMCTS, I saw the following example:

23:39:01 12981570 Stockfish 190826(2): info depth 45 seldepth 75 multipv 1 score cp 113 lowerbound nodes 29825896348 nps 58630104 hashfull 997 tbhits 1454396 time 508713 pv h7h1 12981570 Stockfish 190826(2): info depth 43 currmove h7h1 currmovenumber 1

23:39:07 12988418 Stockfish 190826(2): info depth 45 seldepth 75 multipv 1 score cp 129 lowerbound nodes 30230948523 nps 58636882 hashfull 997 tbhits 1533367 time 515562 pv h7h1 12988419 Stockfish 190826(2): info depth 42 currmove h7h1 currmovenumber 1

23:40:31 13072130 Stockfish 190826(2): info depth 45 seldepth 75 multipv 1 score cp 150 lowerbound nodes 35120560008 nps 58605276 hashfull 997 tbhits 2314258 time 599273 pv h7h1 13072130 Stockfish 190826(2): info depth 41 currmove h7h1 currmovenumber 1

Why are the "currmove" lines reporting depths less than the previous "lowerbound" lines?

Edit: info depth 43 currmove h7h1 currmovenumber 1 is the main thread, while info depth 45 seldepth 75 multipv 1 score cp 113 lowerbound [...] is from the best auxiliary thread. Doesn't explain why Stockfish would skip line 399 in MainThread::search().

noobpwnftw commented 5 years ago

https://github.com/official-stockfish/Stockfish/blob/8fec8834715a440ac18e24e130888c2c60bab352/src/search.cpp#L1254-L1260

In case of a singular extension it may have searched nothing, if a fail low/high is returned does this not cause a not mainThread to infinite loop?

vondele commented 5 years ago

@snicolet currmove uses the actual depth (rootDepth - failedHighCnt) not rootDepth

Alayan-stk-2 commented 5 years ago

Why are the "currmove" lines reporting depths less than the previous "lowerbound" lines?

Normal behaviour. Committed in SF on 25th october 2018.

See this code in search.cpp : Depth adjustedDepth = std::max(ONE_PLY, rootDepth - failedHighCnt * ONE_PLY);

vondele commented 5 years ago

BTW, given the issue we have open here: https://github.com/official-stockfish/Stockfish/issues/2229 and the reply given here https://github.com/msys2/MINGW-packages/issues/5610#issuecomment-511189219 (including issues mentioned there) pointing at potential deadlocks in libwinpthread-1.dll do we know which version of libwinpthread is being used at TCEC, and if any of these bugs in mingw packages could affect us?

d3vv commented 5 years ago

@vondele Does sf at TCEC with pthreads for Windows (not -static pgo build with native threads)???

Alayan-stk-2 commented 5 years ago

It's dated July 5th. I attached it to this message (just rename the .txt to .dll after retrieving it) so it can be tested/compared.

This seems by far the most likely explanation.

libwinpthread-1.txt

vondele commented 5 years ago

@d3vv I don't know how it is produced, and I don't run on windows, so'll need to pass. The build I think is provided in the forum https://groups.google.com/d/msg/fishcooking/PF3dcA8aPh8/Kahr3LufBAAJ but it does include libwinpthread, which is what triggered my remark above.

d3vv commented 5 years ago

@vondele Quick and simple way ask tcec-team about all binaries which was provided them from sf-team.