search

Disservin / fastchess

fastchess is a chess cli tool to run engine vs engine matches
MIT License
90 stars 21 forks source link

PGNs are not in the same order as they are started #550

Open vondele opened 4 months ago

vondele commented 4 months ago

When playing with sequential book order:

$ cat do_match_fastchess.sh 
binary=./cutechess-cli
binary=./fast-chess

for binary in ./cutechess-cli ./fast-chess
do

echo " ============= $binary ============== "

(

$binary         -recover -repeat -games 2 -rounds 5 -tournament gauntlet\
                -pgnout $binary-out.pgn\
                -srand $RANDOM  -resign movecount=3 score=600 -draw movenumber=34 movecount=8 score=20\
                -variant standard -concurrency 4 -openings file=book.epd format=epd order=sequential \
                -engine name=sf_1 tc=10+0.1 depth=10 cmd=./sf_1 dir=.\
                -engine name=sf_2 tc=10+0.1 depth=10 cmd=./sf_2 dir=.\
                -each proto=uci option.Threads=1

) | grep "Finished game" | sort -n -k 3

done

The games in the pgn by cutechess are in order of the book, with fast-chess they are not:

$ cat book.epd 
8/5pk1/5Rp1/2p1p1Pp/1pP1P2P/3P2K1/2P3r1/7r w - - 0 1
1k5r/1p4p1/1p3p2/3Pn3/5B1p/P4P1P/1p4P1/1R4K1 w - - 0 1
6k1/5pn1/3p2p1/p1pP2R1/PpP1r3/1P3R1P/7K/8 b - - 0 1
8/7p/2nk1p2/3B1pp1/R7/P1K1B1P1/r3PP1P/8 b - - 0 1
5Rk1/pp4p1/2pnp3/8/P3P3/1P1nP2K/3N4/2R5 b - - 0 1
$ grep FEN cutechess-cli-out.pgn 
[FEN "8/5pk1/5Rp1/2p1p1Pp/1pP1P2P/3P2K1/2P3r1/7r w - - 0 1"]
[FEN "8/5pk1/5Rp1/2p1p1Pp/1pP1P2P/3P2K1/2P3r1/7r w - - 0 1"]
[FEN "1k5r/1p4p1/1p3p2/3Pn3/5B1p/P4P1P/1p4P1/1R4K1 w - - 0 1"]
[FEN "1k5r/1p4p1/1p3p2/3Pn3/5B1p/P4P1P/1p4P1/1R4K1 w - - 0 1"]
[FEN "6k1/5pn1/3p2p1/p1pP2R1/PpP1r3/1P3R1P/7K/8 b - - 0 1"]
[FEN "6k1/5pn1/3p2p1/p1pP2R1/PpP1r3/1P3R1P/7K/8 b - - 0 1"]
[FEN "8/7p/2nk1p2/3B1pp1/R7/P1K1B1P1/r3PP1P/8 b - - 0 1"]
[FEN "8/7p/2nk1p2/3B1pp1/R7/P1K1B1P1/r3PP1P/8 b - - 0 1"]
[FEN "5Rk1/pp4p1/2pnp3/8/P3P3/1P1nP2K/3N4/2R5 b - - 0 1"]
[FEN "5Rk1/pp4p1/2pnp3/8/P3P3/1P1nP2K/3N4/2R5 b - - 0 1"]
$ grep FEN fast-chess-out.pgn
[FEN "8/5pk1/5Rp1/2p1p1Pp/1pP1P2P/3P2K1/2P3r1/7r w - - 0 1"]
[FEN "8/5pk1/5Rp1/2p1p1Pp/1pP1P2P/3P2K1/2P3r1/7r w - - 0 1"]
[FEN "6k1/5pn1/3p2p1/p1pP2R1/PpP1r3/1P3R1P/7K/8 b - - 0 1"]
[FEN "6k1/5pn1/3p2p1/p1pP2R1/PpP1r3/1P3R1P/7K/8 b - - 0 1"]
[FEN "1k5r/1p4p1/1p3p2/3Pn3/5B1p/P4P1P/1p4P1/1R4K1 w - - 0 1"]
[FEN "1k5r/1p4p1/1p3p2/3Pn3/5B1p/P4P1P/1p4P1/1R4K1 w - - 0 1"]
[FEN "8/7p/2nk1p2/3B1pp1/R7/P1K1B1P1/r3PP1P/8 b - - 0 1"]
[FEN "8/7p/2nk1p2/3B1pp1/R7/P1K1B1P1/r3PP1P/8 b - - 0 1"]
[FEN "5Rk1/pp4p1/2pnp3/8/P3P3/1P1nP2K/3N4/2R5 b - - 0 1"]
[FEN "5Rk1/pp4p1/2pnp3/8/P3P3/1P1nP2K/3N4/2R5 b - - 0 1"]
gahtan-syarif commented 4 months ago

perhaps a fixed nodes test comparison? as the order in which the games are finished affects the order in which the games are listed in the pgn

vondele commented 4 months ago

this is fixed depth, so already deterministic game play (at least should be).

Now, I think the point is, with cutechess the order the games are finished doesn't affect the order in which the games are listed. The games are listed in the order they are started.

Disservin commented 4 months ago

id prefer not doing this as i don't really see and advantage or use case and only downsides (if something goes wrong i.e. pc crash you only have a few pgns depending on when the games finish instead of all leading up to the pc crash) and this would require some bookkeeping for pgns and saving them in memory to be written at a later point

Disservin commented 4 months ago

I'll leave this open for now until this becomes an issue for someone relying on this behaviour

vondele commented 4 months ago

I can give an example where this would be useful.. I now test if all games have the same PlyCount between cutechess and fast-chess, but I can't easily do a 1-to-1 comparison since the order is different. Hence, I sort the PlyCount's before comparison. This might miss some bugs (i.e. two games have different PlyCount, but when sorted it is the same), even though it will probably be few.

I don't think it is critical though, more a nice to have.