test.py not working on Mac OS X

[sshivaji]

The pgn offsets seem to be sorted when compiled by Mac OS X's clang.

I am using a mac for travel. Somehow, these offsets cause issues on the Mac and the tests fail. The offsets I committed work. Puzzled on what the reason can be..

This is reflected by a recent commit where I had to change the pgn offsets json in test.py.

[gbtami]

What Python version are you using to run test.py? I'm asking this because test.py is Python2 only at this moment.

[sshivaji]

Its python 2.7.10. However, the problem is beyond python from my tests. There is different output from the chess_db C++ code and I am not sure why. The pgn offsets are sorted differently.

Ubuntu output (same machine as the chessui deployment):

shiv@drshivaji:~/chess/chessui/chess_db/parser$ ./parser find ../pgn/romero.bin rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
{
    "fen": "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1",
    "key": 5060803636482931868,
    "moves": [
       {
            "move": "e2e4", "weight": 49151, "games": 3, "wins": 1, "losses": 2, "draws": 0, "pgn offsets": [0, 5176, 3656]
       },
       {
            "move": "d2d4", "weight": 16383, "games": 1, "wins": 1, "losses": 0, "draws": 0, "pgn offsets": [1688]
       }
    ]
}

Mac OS X output:

Miyukis-MacBook-Pro:parser shiv$ ./parser find ../pgn/romero.bin rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
{
    "fen": "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1",
    "key": 5060803636482931868,
    "moves": [
       {
            "move": "e2e4", "weight": 49151, "games": 3, "wins": 1, "losses": 2, "draws": 0, "pgn offsets": [0, 3656, 5176]
       },
       {
            "move": "d2d4", "weight": 16383, "games": 1, "wins": 1, "losses": 0, "draws": 0, "pgn offsets": [1688]
       }
    ]
}

[sshivaji]

My current guess (@mcostalba can validate this) is that the entries are written in a different order for the same key in write_poly_file (for clang and gcc). Will post here again after I look for this.

[mcostalba]

Maybe could be a big endian issue. You could make the same pgn file on Mac and on a pc and check if bin files are the same.

[mcostalba]

Ok i understood (maybe). It is due to different std::sort implementation on different platforms. Indeed std::sort is not guaranteed to be implementation independent. We should use std::stable_sort but I fear is much slower.

[gbtami]

I think it's better to leave std::sort in C++ and use sorted(offset_list) in Python code where needed.

[sshivaji]

I will update the test cases to compare pgn offsets in a sorted manner then and submit a PR.

Agree that we don't need stable sort if it makes book building slower.

[mcostalba]

Fixed now.