position-tester

Test your chess engines against different sets of positions for a cheap measure of relative strength.

Data Source

ICCF data: GM correspondence chess positions

Prerequisites

1. Python and a couple libraries
2. Chess engine(s) and weights files (if applicable)

Position Testing files

1. position-tester.py (Python program)
2. Position set (text file with FENs)
3. Configuration file (JSON file with engine path and settings)
4. Networks list (text file with the list of nets to test)

Program Input Syntax

python position-tester.py NetworksList.txt ConfigurationSettings.json OutputSummary.txt OuputLog.log

Output

1. Results summary
2. Detailed log for each position

Sample NetworksList.txt

weights_run1_11248.pb.gz
weights_run1_22154.pb.gz
weights_run2_32930.pb.gz
weights_run1_42482.pb.gz
weights_run2_50778.pb.gz

Sample ConfigurationSettings.json

The parameters EPD and enginePath are mandatory as is weights_path for NN engines. Other parameters are optional.

{
    "EPD" : "PositionSet.txt",
    "enginePath" : "C:/Users/Computer/Downloads/lc0-v0.21.2-windows-cuda/lc0.exe",
    "weights_path" : "C:/Users/Computer/Downloads/lc0-v0.21.2-windows-cuda/",
    "threads" : 2,
    "minibatchsize" : 256,
    "backend" : "cudnn-fp16",
    "nodes" : 10000
}

Sample Position Set file

1b1n2B1/3k3p/p4pp1/1p6/1B6/P6P/1P3PP1/5K2 b - - bm Nc6 ; 61485
1B1q1rk1/pQ2pp1p/4bbp1/3p4/3P4/2r1P1P1/P4PBP/R4RK1 w - - bm Rfb1 ; 8
1b1qr1k1/1p1b1pp1/5n1p/1PP1p3/3p4/3P1NP1/r2B1PBP/1R1QR1K1 w - - bm Qc1 ; 14
1b1qr1k1/1p1r2p1/p2pbp1p/P3nN2/2P1P1P1/1PB4P/5QB1/R4R1K b - - bm Bc7 ; 61490
1b1qr1k1/1p3pp1/5n1p/1PP2b2/3pP3/1R3NP1/r2B1PBP/2Q1R1K1 b - - bm Bxe4 ; 61492
1b1r1rk1/1b3p2/8/Ppp1q1pp/4P3/P2B2N1/2Q2PPP/3R1R1K w - - bm Qe2 ; 27
1b1r1rk1/pb1nqp2/2p4p/1p2p1pn/3PP3/P2B1NB1/1PQ1NPPP/3R1RK1 w - - bm b4 ; 28
1b1r2k1/1b2q2p/p5p1/2p2p2/B1P5/1Q3P1P/PP2NRP1/6K1 w - - bm Kf1 ; 30
1b1r2k1/1b5p/p5p1/2p2p2/B1P4q/5P1P/PP2NRP1/3Q2K1 w - - bm Qb3 ; 32
1b1r2k1/1p5p/3q2pB/Qp1pNp2/P4P2/4n2P/6P1/1R5K b - - bm Nc4 ; 61512
1b1r2k1/1p5p/3q2pB/Qp1pNp2/P4P2/4n2P/6P1/3R3K w - - bm Rb1 ; 45
1b1r2k1/1p5p/3q2pB/Qp1pNp2/P4P2/7P/2n3P1/R6K w - - bm Rd1 ; 47
1B1R4/7k/5pp1/p4r2/Pb3P1p/1B4nP/7K/8 w - - bm Rd7+ ; 66
[...]

Sample Output Summary

network     req_nodes   avg_nodes   agreed  total   percent avg_1st_agree
weights_run1_11248.pb.gz    100 110 38356   80075   47.900  10.331
weights_run1_22154.pb.gz    100 110 34040   80075   42.510  9.927
weights_run2_32930.pb.gz    100 110 38653   80075   48.271  10.990
weights_run1_42482.pb.gz    100 110 40092   80075   50.068  11.292
weights_run2_50778.pb.gz    100 111 34253   80075   42.776  12.240

Sample Ouput Log

#### agree, iccf_moves, nodesUsed, position_id, toPlay,                  pieces Count, weight, mpv 1 move, mpv 1 eval
#### mpv 2 move, mpv 2 eval,  mpv 3 move ; mpv3 eval                 probability (P), count of agree List, agree List
1, gxf4, 10006, 349, W, 28, weights_run2_32930.pb.gz, 0.1507, gxf4, 38, Bh3, 32, Rd6, -17, 2, [[-3,  Bh3, 52], [7,  gxf4, 54]]
1, Nc4, 10045, 629, W, 28, weights_run2_32930.pb.gz, 0.1901, Nc4, 23, fxe5, -28, Kh2, -115, 3, [[4,  Nc4, 17], [-9,  Kh2, -53], [22,  Nc4, 2]]
1, Bf4, 10154, 894, W, 28, weights_run2_32930.pb.gz, 0.29, Bf4, 48, Bd3, 48, Re1, 40, 1, [[2,  Bf4, 55]]
1, Rac1, 10022, 1382, W, 28, weights_run2_32930.pb.gz, 0.1614, Rac1, 43, a3, 24, Nd4, 21, 3, [[3,  Rac1, 32], [-6,  a3, 35], [11,  Rac1, 30]]
1, d5, 10021, 1458, W, 28, weights_run2_32930.pb.gz, 0.153, d5, 33, Re1, 26, h4, 12, 1, [[3,  d5, 49]]
0, Ne3, 10302, 2620, W, 28, weights_run2_32930.pb.gz, 0.0934, Bd1, 71, Ne3, 66, Nc3, 62, 7, [[-4,  Bd1, 47], [141,  Ne3, 62], [-211,  Bd1, 69], [325,  Ne3, 61], [-9376,  Bd1, 71], [9833,  Ne3, 66], [-10302,  Bd1, 71]]
1, Nde3, 10251, 2724, W, 28, weights_run2_32930.pb.gz, 0.2795, Nde3, 229, Qf2, 212, Nf2, 160, 1, [[2,  Nde3, 254]]
[...]