Failing perft tests

[mcepl]

Trying on Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0 and two tests fail

count moves, complex board
Fen: r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq -
    ply 4 expecting 4085603; got 4085675
    4085675 != 4085603 (diff 72)

count moves, in check, promotions
Fen: r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1
    ply 4 expecting 422333; got 422561
    422561 != 422333 (diff 228)

[douglasbagnall]

Ah yes. This refers to these tests: http://p4wn.sourceforge.net/src/auto-test.html

It has been failing forever, and I forgot about it, so thanks for the report.

What these tests do is compare known the number of states reachable at a specified depth from a particular board state with a number known to be correct (known as perft testing https://chessprogramming.wikispaces.com/Perft).

I don't think I came up with a conclusive theory as to why they were failing, or whether the fault was with the testing framework rather than the engine. But some possibilities are:

• Checkmates and stalemates before the specified level are counted differently. Say you are searching to depth 3 and there are n checkmates at depth 2. Do these count as 0 depth 3 states (because search doesn't continue), 1 or 2 depth 3 state (because a win on either side is a game state), or n states, as if each leaf nodes stretches on forever?
• The engine counts leaf node pawn promotions differently. E.g., perft numbers might consider e8=Q to be the same move as e8=N, while p4wn might consider them separately.
• There could be a real bug in p4wn, where it is considering moves that don't exist.

[douglasbagnall]

Fixed in ca70926f4e541a3715e7dc09450b912543178373.

The problem was in the counting, not in the actual move discovery, so this didn't affect real games.