search

Disservin / fast-chess

fast-chess is a command-line tool to run engine vs engine matches in chess.
MIT License
66 stars 18 forks source link

program doesnt send "stop" uci command when ctrl+c is pressed #343

Open gahtan-syarif opened 2 months ago

gahtan-syarif commented 2 months ago

you can notice this when you set the movetime to a large amount like for example st=60, the program would wait for the entire 60 seconds until the engine outputs a bestmove and only then will it save the results. it should instead send a stop command to the engine to force the engine to directly play a bestmove so it can end the tournament.

you can see an example of it in the log file output here:

[03:03:01] <fastchess>Finished tournament.

[03:03:01]  <  3> Engine1 2 ---> info string NNUE evaluation using nn-ae6a388e4a1a.nnue
[03:03:01]  <  3> Engine1 2 ---> info string NNUE evaluation using nn-baff1ede1f90.nnue
[03:03:01]  <  3> Engine1 2 ---> info depth 1 seldepth 3 multipv 1 score cp 78 nodes 39 nps 39000 hashfull 0 tbhits 0 time 1 pv f1g2
[03:03:01]  <  3> Engine1 2 ---> info depth 2 seldepth 5 multipv 1 score cp 78 nodes 83 nps 83000 hashfull 0 tbhits 0 time 1 pv f1g2 b8a6
[03:03:01]  <  3> Engine1 2 ---> info depth 3 seldepth 5 multipv 1 score cp 67 nodes 176 nps 176000 hashfull 0 tbhits 0 time 1 pv f1g2 b8a6 a2a4
[03:03:01]  <  3> Engine1 2 ---> info depth 4 seldepth 5 multipv 1 score cp 67 nodes 227 nps 227000 hashfull 0 tbhits 0 time 1 pv f1g2 b8a6 a2a4
[03:03:01]  <  3> Engine1 2 ---> info depth 5 seldepth 7 multipv 1 score cp 65 nodes 296 nps 148000 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 a2a4 d8c8 h1e1
[03:03:01]  <  3> Engine1 2 ---> info depth 6 seldepth 9 multipv 1 score cp 75 nodes 401 nps 200500 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 h1e1 a6b4 a2a3
[03:03:01]  <  3> Engine1 2 ---> info depth 7 seldepth 8 multipv 1 score cp 75 nodes 471 nps 235500 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 h1e1 a6b4 a2a3 c5c4
[03:03:01]  <  3> Engine1 2 ---> info depth 8 seldepth 9 multipv 1 score cp 78 nodes 616 nps 308000 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4
[03:03:01]  <  3> Engine1 2 ---> info depth 9 seldepth 13 multipv 1 score cp 78 nodes 725 nps 362500 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3
[03:03:01]  <  3> Engine1 2 ---> info depth 10 seldepth 15 multipv 1 score cp 78 nodes 874 nps 437000 hashfull 0 tbhits 0 time 2 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3 e7e6 b2b3
[03:03:01]  <  3> Engine1 2 ---> info depth 11 seldepth 18 multipv 1 score cp 78 nodes 1047 nps 349000 hashfull 0 tbhits 0 time 3 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3 e7e6 b2b3 f6d5 e4d5 g7d4
[03:03:01]  <  3> Engine1 2 ---> info depth 12 seldepth 20 multipv 1 score cp 78 nodes 1504 nps 501333 hashfull 0 tbhits 0 time 3 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3 e7e6 b2b3 f6d5 e4d5 g7d4 c3b5 d4c5
[03:03:01]  <  3> Engine1 2 ---> info depth 13 seldepth 25 multipv 1 score cp 76 nodes 2602 nps 650500 hashfull 0 tbhits 0 time 4 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3 e7e6 b2b3 f6d5 e4d5 g7d4 c3b5 d4c5 b3c4 d3c1 b1c1 e6d5 d1d5 c8c6
[03:03:01]  <  3> Engine1 2 ---> info depth 14 seldepth 25 multipv 1 score cp 76 nodes 3209 nps 641800 hashfull 0 tbhits 0 time 5 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 f3d4 a6b4 h1d1 b4d3 h2h3 e7e6 b2b3 f6d5 e4d5 g7d4 c3b5 d4c5 b3c4 d3c1 b1c1 e6d5 d1d5 c8c6
[03:03:01]  <  3> Engine1 2 ---> info depth 15 seldepth 26 multipv 1 score cp 75 nodes 18116 nps 787652 hashfull 10 tbhits 0 time 23 pv f1g2 b8a6 a2a4 d8c8 a1a3 a8b8 d1e2 b8b4 e4e5 d6e5 f3e5 c8a8
[03:03:01]  <  3> Engine1 2 ---> info depth 16 seldepth 23 multipv 1 score cp 71 nodes 41652 nps 785886 hashfull 21 tbhits 0 time 53 pv f1g2 b8a6 a2a4 d8c8 a1a3 a8b8 h1e1 a6c7 d1c2 b8b4 h2h4
[03:03:01]  <  3> Engine1 2 ---> info depth 17 seldepth 23 multipv 1 score cp 69 nodes 97775 nps 757945 hashfull 46 tbhits 0 time 129 pv f1g2 b8a6 a2a4 d8c8 a1a3 a6b4 c1g5 c5c4 d1e2 h7h6 g5d2 b4d3 f3d4 e7e6
[03:03:01]  <  3> Engine1 2 ---> info depth 18 seldepth 22 multipv 1 score cp 66 nodes 176609 nps 764541 hashfull 81 tbhits 0 time 231 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 a2a3 a6c5 e4e5 d6e5 f3e5 e7e6 d5e6 c8b7 f2f3 c5e6 c1e3 f8e8 e5c4
[03:03:01]  <  3> Engine1 2 ---> info depth 19 seldepth 24 multipv 1 score cp 67 nodes 190958 nps 769991 hashfull 88 tbhits 0 time 248 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 a2a3 a6c5 e4e5 d6e5 f3e5 c5a4 e2f3 e7e6 d5d6
[03:03:01]  <  3> Engine1 2 ---> info depth 20 seldepth 24 multipv 1 score cp 69 nodes 242328 nps 771745 hashfull 110 tbhits 0 time 314 pv f1g2 b8a6 a1b1 c5c4 d1e2 d8c8 h1d1 a6b4 a2a3 b4d3 f3e1 d3e1 d1e1 f6d7 c3a2 f7f5 a2b4 d7e5 f2f4
[03:03:01]  <  3> Engine1 2 ---> bestmove f1g2 ponder b8a6
[03:03:01] <fastchess>Saved results.

[03:03:01]  <  1> Engine2 <--- quit

[03:03:01]  <  1> Engine1 <--- quit

after i press ctrl+c and it outputs finished tournament, it lets the engine still run and only when the engine outputs a bestmove does it save results and quit.

Disservin commented 2 months ago

this sounds pretty reasonable to do

Disservin commented 2 months ago

nvm i think we basically need check the global stop in the readProcess.

vondele commented 2 weeks ago

I think I have a related observation. If one engine crashes / aborts, the other processes appear to remain active (even though fast-chess quits), and I need to kill them by hand.

Disservin commented 2 weeks ago

I think I have a related observation. If one engine crashes / aborts, the other processes appear to remain active (even though fast-chess quits), and I need to kill them by hand.

What I was able to reproduce is that when an engine crashes the error propagated back to fastchess and wasn't handled gracefully, so fastchess also threw and didn't cleanup properly. Should be now fixed https://github.com/Disservin/fast-chess/commit/9ac8a968a231efcbbb37238d35eb7b9a5109a052, the current implementation will try to continue playing with the remaining available engines until all games are played or all engines have crashed. In either case all the processes should have been stopped.

vondele commented 2 weeks ago

actually thinking about more about https://github.com/Disservin/fast-chess/commit/9ac8a968a231efcbbb37238d35eb7b9a5109a052 I'm not sure that this is the best solution. Stopping the match might be a better solution, probably even exiting with a non-zero exit code, unless -recover is specified, in which case the engine should be restarted, I think. I think the current solution might too easily lead to the engine crash going unnoticed.

Disservin commented 2 weeks ago

Well the patch first fixes something which shouldn't happen, expected behavior is something else.

I think I'll first reimplement the -recover option and then make the tournament exit when not specified. One thing I'm unsure about -recover is: Does it restart the engine at the point in the match where it crashed and continues with that position or is the matchup for the specific opening replayed? Also what happens when the engine keeps crashing?

gahtan-syarif commented 2 weeks ago

Well the patch first fixes something which shouldn't happen, expected behavior is something else.

I think I'll first reimplement the -recover option and then make the tournament exit when not specified. One thing I'm unsure about -recover is: Does it restart the engine at the point in the match where it crashed and continues with that position or is the matchup for the specific opening replayed? Also what happens when the engine keeps crashing?

its this i think https://github.com/cutechess/cutechess/blob/1071d84cf272bd7deca0964336bf02e367e2b22b/projects/lib/src/tournament.h#L181

so basically we already have -recover on by default since if an engine disconnects it would just resume the tournament (and keep disconnecting), if -recover is not specified then whole tournament stops the moment a crash happens https://github.com/cutechess/cutechess/blob/1071d84cf272bd7deca0964336bf02e367e2b22b/projects/lib/src/tournament.cpp#L781

Disservin commented 2 weeks ago

so basically we already have -recover on by default since if an engine disconnects it would just resume the tournament

Well currently we are losing engines after a crash, so if you have as n crashes and n concurrency, the tournament won't continue because there are no more engines.

So with recover the match is just annotated as lost and it continues with the next game OK, can implement that.

gahtan-syarif commented 2 weeks ago

so basically we already have -recover on by default since if an engine disconnects it would just resume the tournament

Well currently we are losing engines after a crash, so if you have as n crashes and n concurrency, the tournament won't continue because there are no more engines.

So with recover the match is just annotated as lost and it continues with the next game OK, can implement that.

yeah, exactly, and have it that with no -recover the tournament finishes, saves, and cleanly exits after the disconnect game

vondele commented 2 weeks ago

So with recover the match is just annotated as lost and it continues with the next game OK, can implement that.

The game is annotated as a loss, and a new engine instance starts the next game.

vondele commented 2 weeks ago

what's the cutechess exit code after an engine crash and no -recover ? if this is a non-zero exit code I could use that... (full disclosure, I'd like to add a CI test to SF where we play a couple of games with an SF compiled with debug=yes so that we can catch things triggering an assert in game play, but not our testing).

Disservin commented 2 weeks ago

https://github.com/Disservin/fast-chess/pull/463

this should cover the recover option and it's associated behavior, if there is a particular need for a non zero exit code i could add that as well