Book find returns different move from book created with/without "full"

[gbtami]

koltanowski.zip The 11. game Koltanowski, George vs. Thompson, Brett after move 9... a6 is 10. Nxe7

"find" on book created with "full" is Ok:

tamas@tami:~/PGN$ parser book koltanowski.pgn full

Processing...done
Sorting...done
Writing Polygot book...done

Games: 14
Moves: 735
Incorrect moves: 0
Unique positions: 90%
Games/second: 14000
Moves/second: 735000
MBytes/second: 9.149
Size of index file (bytes): 11760
Book file: koltanowski.bin
Processing time (ms): 1

tamas@tami:~/PGN$ parser find koltanowski.bin r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10

{
    "fen": "r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10",
    "key": 3452451395811852,
    "moves": [
       {
            "move": "f5e7", "weight": 1, "games": 1, "wins": 1, "losses": 0, "draws": 0, "pgn offsets": [7352]
       }
    ]
}

"find" on book created without "full" seems wrong:

tamas@tami:~/PGN$ parser book koltanowski.pgn

Processing...done
Sorting...done
Writing Polygot book...done

Games: 14
Moves: 735
Incorrect moves: 0
Unique positions: 90%
Games/second: 14000
Moves/second: 735000
MBytes/second: 9.149
Size of index file (bytes): 10880
Book file: koltanowski.bin
Processing time (ms): 1

tamas@tami:~/PGN$ parser find koltanowski.bin r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10

{
    "fen": "r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10",
    "key": 3452451395811852,
    "moves": [
       {
            "move": "g4e6", "weight": 1, "games": 1, "wins": 1, "losses": 0, "draws": 0, "pgn offsets": [8336]
       }
    ]
}

[sshivaji]

I debugged this a little and found the reason. The problem is that if NO MOVE is found, the last entry is returned instead of MOVE_NONE. Interestingly, we can also reproduce by trying to do a find on a fen that is clearly not in the DB. Relevant code is attached below.

However, beyond this, the 2nd issue might be why the position is not in the DB when running in non full mode.

(xenial)shiv@localhost:~/chess/chess_db/parser$ ./parser find ../pgn/koltanowski-full.bin rnbqkbnr/ppppppp1/8/8/8/8/PPPPPPPP/RNBQKBNR w kq
ofs: 752
e.move: 788
{
    "fen": "rnbqkbnr/ppppppp1/8/8/8/8/PPPPPPPP/RNBQKBNR w kq - 0 1",
    "key": 1179851944398274625,
    "moves": [
       {
            "move": "e2e3", "weight": 1, "games": 2, "wins": 2, "losses": 0, "draws": 0, "pgn offsets": [3992, 4528]
       }
    ]
}

I compared the logic in this codebase compared to Stockfish's book.probe, looks like in Stockfish we are parsing the move and returning MOVE_NONE if the move is not legal. Here we are returning the last entry instead.

/// probe() tries to find a book move for the given position. If no move is
/// found, it returns MOVE_NONE. If pickBest is true, then it always returns
/// the highest-rated move, otherwise it randomly chooses one based on the
/// move score.

size_t PolyglotBook::probe(Key key, const string& fName) {

  if (fileName != fName && !open(fName.c_str()))
      return 0;

  size_t ofs = find_first(key);
  close();
  return ofs;
}

/// find_first() takes a book key as input, and does a binary search through
/// the book file for the given key. Returns the index of the leftmost book
/// entry with the same key as the input.

size_t PolyglotBook::find_first(Key key) {

  seekg(0, ios::end); // Move pointer to end, so tellg() gets file's size

  size_t low = 0, mid, high = (size_t)tellg() / SizeOfPolyEntry - 1;
  PolyEntry e;

  assert(low <= high);

  while (low < high && good())
  {
      mid = (low + high) / 2;

      assert(mid >= low && mid < high);

      seekg(mid * SizeOfPolyEntry, ios_base::beg);
      *this >> e;

      if (key <= e.key)
          high = mid;
      else
          low = mid + 1;
  }

  assert(low == high);

  return low * SizeOfPolyEntry;
}

[mcostalba]

@sshivaji thx! It should be fixed now. Please confirm if it works for you.

[sshivaji]

If a position is not found, it is reported correctly. However, for some reason the "full" mode has a move in it but the non "full" mode does not in koltanowski.pgn.

In the non full book (no game_ids), I get

shiv@shiv-Inspiron-3847:~/chess/sshivaji/chess_db/parser$ parser find koltanowski.bin r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10
 {
    "fen": "r1bqk2r/1pppbppp/p1n1n3/4PN2/2B5/8/PPP2PPP/RNBQR1K1 w kq - 0 10",
    "key": 3452451395811852,
    "moves": [
    ]
}