search

fsmosca / chess-artist

A python script that can annotate chess games in pgn file with static evaluation or search score of an engine, can annotate an epd file with acd, acs, bm, and ce opcodes, can test engine with epd test suite and can generate chess puzzles.
GNU General Public License v3.0
69 stars 18 forks source link

create puzzles/test files stuck #81

Closed Kraulfisch closed 3 years ago

Kraulfisch commented 3 years ago

Hello Super program, thank you very much! But when I want to create test files from a larger number of games in a pgn (20'000 games), it analyzing the fen's, but suddenly it is just stuck at one fen. image The command I run is: "C:\Users\Raoul Sidler\Downloads\chess-artist\chess-artist.exe" --infile "C:\Users\Raoul Sidler\Downloads\chess-artist-master\chess-artist-master\PGN\Carlsen.pgn" --outfile "C:\Users\Raoul Sidler\Downloads\chess-artist-master\chess-artist-master\PGN\out_Carlsen.pgn" --enginefile "C:\Users\Raoul Sidler\Downloads\chess-artist-master\chess-artist-master\Engine\stockfish\stockfish_12_x64_modern" --engineoptions "Hash value 128" --movestart 8 --movetime 10000 --job createpuzzle --eval search

I'm not really sure about the output file, cause in the test command it is just the input with "out_", but it should be saved in puzzle.epd, so I tried both of this, but both stopped at the same fen. I don't know if it is my pgn file, or if the programm is over, but there aren't any new puzzles in the puzzle.epd file...

Thanks a lot!

fsmosca commented 3 years ago

I see that you are using the old version. The latest version is not yet compiled to exe file.

Regarding the output puzzle.epd, in the old version generating puzzle is rare because of the algorithm in it. In the latest version chess_artist.py it is not too strict and there is a -puzzle-score-margin to play around.

Regarding the program getting stuck, I have no idea for it. If you send me the pgn file or the exact game in the pgn file where it get stuck, I can test it and resolve the issue.

Thanks for raising the issue.

Kraulfisch commented 3 years ago

Hello Thanks for the response. Which one is the newer version? I'll give it a try. This is the pgn file I use: Carlsen.txt

fsmosca commented 3 years ago

Which one is the newer version? I'll give it a try.

The chess_artist.py file in this link: https://github.com/fsmosca/chess-artist

This is the pgn file I use: Carlsen.txt

Thanks I will test it.

fsmosca commented 3 years ago

All right I found the main issue. When the score of the position is winning or losing by some margin after some iteration, I go to next game without sending the stop command to the engine.

This is the fix, and improvement in logging, now stop command is sent to engine, so that it will stop searching.

2021-03-24 13:13:38,664 :: pid: 16564 :: tid: 9924 :: INFO :: gameNum: 1
2021-03-24 13:13:38,671 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> ucinewgame
2021-03-24 13:13:38,671 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> position fen 3r1rk1/pp1nqppp/2pbpn2/3p4/2PP4/1PNQPN2/PB3PPP/3R1RK1 w - - 7 12
2021-03-24 13:13:38,671 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> go movetime 2000
2021-03-24 13:13:38,719 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 1 seldepth 1 multipv 1 score cp 10 nodes 59 nps 59000 tbhits 0 time 1 pv e3e4
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 2 seldepth 2 multipv 1 score cp 344 nodes 133 nps 133000 tbhits 0 time 1 pv c4c5 a7a6 c5d6
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: read next game, the absolute score of 344 > score threshold of 200
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> stop
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 3 seldepth 3 multipv 1 score cp 344 nodes 182 nps 182000 tbhits 0 time 1 pv c4c5 a7a6 c5d6
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 4 seldepth 4 multipv 1 score cp 296 nodes 300 nps 150000 tbhits 0 time 2 pv c4c5 d6c5 d4c5 d7c5
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 5 seldepth 5 multipv 1 score cp 367 nodes 346 nps 173000 tbhits 0 time 2 pv c4c5 d6c5 d4c5
2021-03-24 13:13:38,720 :: pid: 16564 :: tid: 9924 :: DEBUG :: << bestmove c4c5 ponder d6c5
2021-03-24 13:13:38,725 :: pid: 16564 :: tid: 9924 :: INFO :: gameNum: 2
2021-03-24 13:13:38,732 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> ucinewgame
2021-03-24 13:13:38,732 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> position fen r2q1rk1/ppp1bppp/1nn1p3/3pP2b/3P1B2/2P2N1P/PP1NBPP1/R2Q1RK1 w - - 0 12
2021-03-24 13:13:38,732 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> go movetime 2000
2021-03-24 13:13:38,768 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 1 seldepth 1 multipv 1 score cp 39 nodes 36 nps 18000 tbhits 0 time 2 pv g2g4
2021-03-24 13:13:38,768 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 2 seldepth 2 multipv 1 score cp 287 nodes 87 nps 43500 tbhits 0 time 2 pv g2g4 f7f6 e5f6
2021-03-24 13:13:38,768 :: pid: 16564 :: tid: 9924 :: DEBUG :: read next game, the absolute score of 287 > score threshold of 200
2021-03-24 13:13:38,768 :: pid: 16564 :: tid: 9924 :: DEBUG :: >> stop
2021-03-24 13:13:38,769 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 3 seldepth 3 multipv 1 score cp 287 nodes 130 nps 65000 tbhits 0 time 2 pv g2g4 f7f6 e5f6
2021-03-24 13:13:38,769 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 4 seldepth 4 multipv 1 score cp 138 nodes 443 nps 221500 tbhits 0 time 2 pv g2g4 g7g5 g4h5 g5f4
2021-03-24 13:13:38,769 :: pid: 16564 :: tid: 9924 :: DEBUG :: << info depth 5 seldepth 5 multipv 1 score cp 178 nodes 501 nps 250500 tbhits 0 time 2 pv g2g4
2021-03-24 13:13:38,769 :: pid: 16564 :: tid: 9924 :: DEBUG :: << bestmove g2g4 ponder g7g5

Other fix to minor issue is to use time check with high resolution, as this algorithm is checking the search score of the engine while it is searching.

I will update the main chess_artist.py in the repository and compile a new chess_artist.exe file after some tests.

fsmosca commented 3 years ago

Another improvement is to wait a little bit more at least depth 8 or movetime/16 has been consumed before deciding if position is winning or lossing, this would increase the probability that our decision is correct to exclude this position as its score is really beyond the score threshold.

In the example below, stop was sent after depth 10.

2021-03-24 13:44:50,777 :: pid: 13640 :: tid: 12952 :: DEBUG :: >> ucinewgame
2021-03-24 13:44:50,777 :: pid: 13640 :: tid: 12952 :: DEBUG :: >> position fen r1b2k1r/p3q3/1pn2p2/2pp1p1p/3P3P/P1P1B1Q1/2P2PP1/R3K1NR w KQ - 2 16
2021-03-24 13:44:50,777 :: pid: 13640 :: tid: 12952 :: DEBUG :: >> go movetime 8000
2021-03-24 13:44:50,808 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 1 seldepth 1 multipv 1 score cp 155 nodes 43 nps 43000 tbhits 0 time 1 pv g1e2
2021-03-24 13:44:50,808 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 2 seldepth 2 multipv 1 score cp 238 nodes 81 nps 81000 tbhits 0 time 1 pv g1e2 c5d4 c3d4
2021-03-24 13:44:50,809 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 3 seldepth 3 multipv 1 score cp 238 nodes 133 nps 133000 tbhits 0 time 1 pv g1e2 c5d4 c3d4
2021-03-24 13:44:50,809 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 4 seldepth 4 multipv 1 score cp 328 nodes 194 nps 194000 tbhits 0 time 1 pv g1e2
2021-03-24 13:44:50,810 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 5 seldepth 5 multipv 1 score cp 98 nodes 1106 nps 553000 tbhits 0 time 2 pv g1e2 c8a6 g3f3 a6e2 f3e2
2021-03-24 13:44:50,811 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 6 seldepth 7 multipv 1 score cp 77 nodes 2278 nps 569500 tbhits 0 time 4 pv d4c5 h8g8 g3f3 c6e5 f3d5
2021-03-24 13:44:50,812 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 7 seldepth 7 multipv 1 score cp 117 nodes 2527 nps 631750 tbhits 0 time 4 pv d4c5 b6c5 g1e2 f8f7
2021-03-24 13:44:50,814 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 8 seldepth 9 multipv 1 score cp 199 nodes 3686 nps 614333 tbhits 0 time 6 pv g1e2 h8g8 g3f3
2021-03-24 13:44:50,822 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 9 seldepth 10 multipv 1 score cp 175 nodes 9044 nps 602933 tbhits 0 time 15 pv g1e2 c5d4 c3d4 c8a6 e2f4 e7e8 f4g6 f8g8 g6h8 g8h8
2021-03-24 13:44:50,837 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 10 seldepth 15 multipv 1 score cp 206 nodes 17169 nps 592034 tbhits 0 time 29 pv g1e2 h8g8 g3f3 e7f7 e1c1 c5d4 c3d4 c6e5
2021-03-24 13:44:50,837 :: pid: 13640 :: tid: 12952 :: DEBUG :: read next game, the absolute score of 206 > score threshold of 200
2021-03-24 13:44:50,837 :: pid: 13640 :: tid: 12952 :: DEBUG :: >> stop
2021-03-24 13:44:50,837 :: pid: 13640 :: tid: 12952 :: DEBUG :: << info depth 11 seldepth 15 multipv 1 score cp 206 nodes 17360 nps 578666 tbhits 0 time 30 pv g1e2 h8g8 g3f3 e7f7 e1c1 c5d4 c3d4 c6e5
2021-03-24 13:44:50,837 :: pid: 13640 :: tid: 12952 :: DEBUG :: << bestmove g1e2 ponder h8g8
fsmosca commented 3 years ago

Update with commit https://github.com/fsmosca/chess-artist/commit/1081a8f31e7d60e9e8925d79bdb18c2c74dd21b2 to address this issue.

Try this chess_artist.py, I tested it with your pgn file and so far there is problem.

game number: 81
pos number: 1
pos number: 2
pos number: 3
pos number: 4
pos number: 5
pos number: 6
pos number: 7
pos number: 8
pos number: 9
pos number: 10
pos number: 11
pos number: 12
pos number: 13
pos number: 14
pos number: 15
analyzing fen r3kb1r/pp1qpppp/2np1n2/8/3pP3/2P2N2/PP3PPP/RNBQ1RK1 w kq - 0 8 ...
pos number: 16
analyzing fen r3kb1r/pp1qpppp/2np1n2/8/3PP3/5N2/PP3PPP/RNBQ1RK1 b kq - 0 8 ...
pos number: 17
analyzing fen r3kb1r/pp1qpppp/2n2n2/3p4/3PP3/5N2/PP3PPP/RNBQ1RK1 w kq - 0 9 ...
pos number: 18
analyzing fen r3kb1r/pp1qpppp/2n2n2/3pP3/3P4/5N2/PP3PPP/RNBQ1RK1 b kq - 0 9 ...
pos number: 19
...

The output would look like this.

3r1rk1/pp1nqppp/2pbpn2/3p4/2PP4/1PNQPN2/PB3PPP/3RR1K1 b - - bm Bb4; Ubm d6b4; sm dxc4; Ae "Stockfish 12";
6k1/p4ppp/2p2n2/1p6/2P5/4PP2/PN4PP/6K1 b - - bm Kf8; Ubm g8f8; sm Kf8; Ae "Stockfish 12";
5k2/p4ppp/2p2n2/1p6/2P5/4PP2/PN4PP/6K1 w - - bm e4; Ubm e3e4; sm Kf2; Ae "Stockfish 12";
r4bk1/4nppp/4p3/p1rpP3/n4B1P/PRP3P1/2N2P2/RN3K2 b - - bm Rcc8; Ubm c5c8; sm Rac8; Ae "Stockfish 12";

The sm opcode value is the actual move in the game.

Kraulfisch commented 3 years ago

Hello Thanks for the response. I'm not such a pro like you and I'm not really used to GitHub. How do I run the chess_artist.py file? I didn't found any input file or something like that in the code.

Sorry for the probably really stupid questions, I just started with programming neuronal networks and not really into getting the right data. I just learned with data I got for training.

fsmosca commented 3 years ago

I am using windows 10.

Kraulfisch commented 3 years ago

Hey My system don't recognize "git". 'git' is not recognized as an internal or external command, operable program or batch file.

I have python installed with anaconda, but this shouldn't be a problem. Should I clone it with GitHub desktop?

Kraulfisch commented 3 years ago

Another improvement is to wait a little bit more at least depth 8 or movetime/16 has been consumed before deciding if position is winning or lossing, this would increase the probability that our decision is correct to exclude this position as its score is really beyond the score threshold.


Just tried it, also with depth 25 it is stuck at the same position. Seems like the position isn't winning.
fsmosca commented 3 years ago

Hey My system don't recognize "git". 'git' is not recognized as an internal or external command, operable program or batch file.

I have python installed with anaconda, but this shouldn't be a problem. Should I clone it with GitHub desktop?

GitHub desktop should be fine.

fsmosca commented 3 years ago

Another improvement is to wait a little bit more at least depth 8 or movetime/16 has been consumed before deciding if position is winning or lossing, this would increase the probability that our decision is correct to exclude this position as its score is really beyond the score threshold.

Just tried it, also with depth 25 it is stuck at the same position. Seems like the position isn't winning.

Did you use the latest chess_artist.py?

What game number is the problem or what position number?

On your command line add --log flag. Then see the chess_artist_log.txt file.

Example: python chess_artist.py ... --job createpuzzle --eval search --log

Kraulfisch commented 3 years ago

Okay, I‘m gonna try that. I will be able to test it on mondy, because I‘m bot at home now. I will post if it works or not.

Kraulfisch commented 3 years ago

Just tried it with the chess_artist.py. Tested with this code,:

C:\Users\Raoul Sidler\Desktop\Schule\maturaarbeit\test_files\chess-artist>chess_artist.py --infile ./PGN/skillingopp20.pgn --outfile out_skilling.pgn --enginename "Sf12" --enginefile ./Engine/stockfish/stockfish_12_x64_modern.exe --engineoptions "Threads value 1, Hash value 256" --movestart 12 --movetime 10000 --job createpuzzle --eval search --log

Then my editor (atom) opens. If I close it, I get the following error code:

Attempting to call a function in a renderer window that has been closed or released. Function provided here: Object. (C:\Users\Raoul Sidler\AppData\Local\atom\app-1.55.0\resources\app.asar\node_modules\github\lib\worker.js:79:22 Remote event names: crashed,

Same happens when I try to run the batch file from the clone.

fsmosca commented 3 years ago

I am not sure what your problem is. chess_artist does not open an editor.

Could you show the log file generated by chess_artist, let's see what happened?

Kraulfisch commented 3 years ago

yes, I was wondering too. But there is no log file. It should be at the command promt, shouldn't it? image

When I run the code or the batch file, a new atom window opens. image

fsmosca commented 3 years ago

Try this:

  1. Open your windows explorer
  2. Locate the chess artist folder and double click the create_puzzle.bat
Kraulfisch commented 3 years ago

Yeah, tried this too. Then this windows open and the atom windows opens: image

Try now to fix this.

Edit: Okay, I know now why the editor opens: It always opens the chess_artist.py file. Seems like the command doesn't recognize the

--infile ./PGN/skillingopp20.pgn --outfile out_skilling.pgn --enginename "Sf12" --enginefile ./Engine/stockfish/stockfish_12_x64_modern.exe --engineoptions "Threads value 1, Hash value 256" --movestart 12 --movetime 10000 --job createpuzzle --eval search --log

I'm not really into command promt etc, so is there any way to give the parameters straight to the chess_artist.py file?

fsmosca commented 3 years ago

Try this, lets see if python can be seen.

  1. Open command prompt
  2. type python --version
fsmosca commented 3 years ago

I'm not really into command promt etc, so is there any way to give the parameters straight to the chess_artist.py file?

No, chess artist is a command line tool.

Kraulfisch commented 3 years ago

Try this, lets see if python can be seen.

  1. Open command prompt
  2. type python --version

Python 3.8.5 image

fsmosca commented 3 years ago

All right. Edit the create_puzzle.bat (right-click the file and press edit), you can open it with notepad then use the command line below. Or just add python in front. Save and double-click the bat file to run.

python chess_artist.py --infile ./PGN/skillingopp20.pgn --outfile out_skilling.pgn --enginename "Sf12" --enginefile ./Engine/stockfish/stockfish_12_x64_modern.exe --engineoptions "Threads value 1, Hash value 256" --movestart 12 --movetime 10000 --job createpuzzle --eval search --log
Kraulfisch commented 3 years ago

Hey, thanks soo much! It works fine now. But there are some questions: The programm is running really slowly and don't uses the full cpu usage. Is this normal? (I'm right now at game 26 after about 2 hours of running. I want to use files with over 2 million games.)

The programm doesn't save the Fen in puzzle.epd. I see the message, but the file doesn't change. Does it save these at the end of the programm? Then it would be better to do smaller files to avoid data loss when your Pc crashes.

Edit: I'm not really sure, but is the Thread value the numbers of threads from the cpu? I have a Ryzen 7 2700x with 8cores/16threads. Should I put the value to 16? or 8?

fsmosca commented 3 years ago

Good, the --movetime 10000 is the analysis time per position, 10000 is in milliseconds so that is 10 seconds.

The position is saved, you can open it with notepad++ for example. They are saved one at a time so no issue with crashes.

Thread is the thread used by engine.

I have not added yet the concurrency support. If you want to use more threads you can divide your pgn files into 12 for example, place each pgn into different folders along with chess_artist.py then you can run individually. In this case since you are using 12 threads just use 1 thread for the engine. You still have 16-12 or 4 threads extra.

Kraulfisch commented 3 years ago

The position is saved, you can open it with notepad++ for example. They are saved one at a time so no issue with crashes.

Good, but where? I can't find it at the puzzle.epd file. image For example, this fen isn't saved in the puzzle.epd image The wacnew.epd contains more fens, but not the one analysed.

fsmosca commented 3 years ago

Close and open again it should be there.

Use notepad++, as it does not need to be closed and opened. The output will be updated

Kraulfisch commented 3 years ago

No, this doesn't work. But there are now 300 fens in the wacnew.epd file, but I can't find oen of the saved ones.

fsmosca commented 3 years ago

wacnew.epd has nothing to do with puzzle.epd.

With this command line:

python chess_artist.py --infile ./PGN/skillingopp20.pgn --outfile out_skilling.pgn --enginename "Sf12" --enginefile ./Engine/stockfish/stockfish_12_x64_modern.exe --engineoptions "Threads value 1, Hash value 256" --movestart 12 --movetime 10000 --job createpuzzle --eval search --log

It will generate a puzzle saved in puzzle.epd. It has nothing to do with wacnew.epd.

Kraulfisch commented 3 years ago

yes, thats exactely the code I use to test it first. But there are no new fens in the puzzle.epd file. I'm nearly done with the skillingopp20.pgn file, problably they are displayed then.

Kraulfisch commented 3 years ago

Oops, my mistake. I opened the wrong puzzle.epd file, not the one directely in the chess-artist folder, but the one in the EPD folder. The fens are saved correctely!!

Thanks again for the very very useful programm and the excellent support you gave to me!!!!

fsmosca commented 3 years ago

It is nice it worked for you.

Later I will implement a new puzzle algorithm, I have no time for this at the moment.