v0.6.1 : remarks and advise

[tissatussa]

Congrats with the new v0.6.1 ! It runs fine in CuteChess, showing its search info in the concerning pane, but i have some remarks and advise.

I compiled Catto with searchDepth 4, lmrFullDepth 4 and lmrMaxReduction 3, which are your defaults, then i put the created binary (44.3 Mb in size) together with the file catto.config.js in one folder, and i added Catto to the CuteChess engine list. In THAT config file i changed those numbers into 5,5,4 - expecting Catto to read this config at initialisation when playing a game, but CuteChess showed search depth is 4, not 5 .. I discovered that the config file is only used when compiling, not when playing : Catto can even play without a config file in its folder, it seems to just use the config settings of the compilation. This is not normal, and you don't mention this functioning in the README, the average user will suspect the config file is needed in the folder.

You should change the way the search parameters are read, by using the UCI set option functionality and remove that config file. Then the uci command would show something like this :

id name Catto v0.?.?
id author nguyenphuminh
option name searchDepth type spin default 4 min 1 max 6 
option name lmrFullDepth type spin default 4 min 1 max 6 
option name lmrMaxReduction type spin default 3 min 1 max 5 

This way Catto will have its default parameter values and the user can adjust those in the "Advanced" tab of the CuteChess "Configure Engine" pane and they will be saved and read by Catto each time it plays a game. This way we can also create another Catto version in CuteChess, having different settings, using the same binary. The version number can be fixed, don't make it a UCI option.

Besides that, the config option 'fen' is not implemented properly : you should create the UCI command position fen .. which is used by CuteChess to set a startposition for the game. With current version v0.6.1 we can not set a starting position in CuteChess.

I also have some remarks about the output.

This screenshot shows that Catto v0.6.1 only displays the last PV line, not the ones before that, like the opponent engine does (and many others). And that last PV line is displayed only after bestmove is given, not during the calculation process .. i like to see Catto changing its mind for bestmove while evaluating a position, displaying that info during play at each depth.

The CuteChess evaluation pane also shows NPS, Nodes Per Second, and the total amount of nodes searched. Here, the opponent engine "Elf v1.30" searches 90 times (!) more nodes (113000 / 1261) then Catto and easily reaches depth 8 or more - which is still low compared to many other engines .. is TypeScript that slow ? btw. setting depth 6 is much to high for Catto to play a game : it doesn't seem to anticipate the remaining time and it will lose the game on time with that setting .. the Elf engine seems to use max 1 / 30 of the remaining time for each move, which is a good practical setting, used by many other engines .. probably Elf can stop its calculations at any moment, when that time's up, and return its last bestmove. This is the right approach.

[nguyenphuminh]

Thanks for the issue!

I discovered that the config file is only used when compiling, not when playing

I think because the file is js code, not json, I think people would expect it to be used only when compiling, but sure I will add some notes for this.

You should change the way the search parameters are read, by using the UCI set option functionality and remove that config file.

Added to my list of todos.

Besides that, the config option 'fen' is not implemented properly : you should create the UCI command position fen .. which is used by CuteChess to set a startposition for the game. With current version v0.6.1 we can not set a starting position in CuteChess.

I already implemented position fen, I think the problem is that currently the uci client takes everything after position fen as a fen string, so something like position fen rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1 moves d2d4 would break, will include a fix for this in the next patch. The fen option is not used for anything really, I will remove it soon.

is TypeScript that slow ?

It is definitely much slower than C or Rust, but I think that's just one factor. Chess.js is also slow, and my code is probably not optimized too.

This screenshot shows that Catto v0.6.1 only displays the last PV line...

When I add iterative deepening (which is in my list of todos), I will go through each depth and logs out info of each depth too. Proper time control will come with iterative deepening as well.

Thanks again! This has been really helpful!

[tissatussa]

Thanks again! This has been really helpful!

i get notifications of many GitHub engines and since i discovered Catto, a new-kid-on-the-block for me, i follow your releases and try to compile and use them. In this case the engine is rather weak, so as a human i'm able to play against it and sometimes i defeat Catto, but it's hard, even with searchDepth 4 .. are you a (good) chess player ? My (Dutch) club rating is about 1900, on LiChess 15m10s Rapid i have around 2000 for some time now.

here's a recent game with Catto, i play the Black pieces. I opened the game with the Dutch Defence (1...f5), claiming square e4 and hoping White wouldn't play 2.e4!? which is the "Staunton Gambit" with lots of tactics .. indeed Catto didn't occupy e4 that way, it played a unique move : 2.d5!? So i had to adjust my setup completely, leading to a nice game which i lost in the end.

Replay the game

regarding the reached search depth and the amount of evaluated nodes : i guess your evaluation logic is still primitive .. i know, i once tried to create my own engine, using the TSCP code, just to learn things and be amazed .. my engine reached max depth 5 in practical play and it would easily lose on time .. strong engines reach depth 20 in a few seconds, often even with many pieces on the board, they must have a smart pruning mechanism, thus skipping many nodes which have "no promise", but maybe considered later when there's time. When playing a serious game, the engine should be able to somehow judge if a position is worth examining for a longer time, because bestmove can be decisive at that moment. When the reached depth is low, decent time management can hardly exist.

i advise you to first properly implement the most common UCI commands, so Catto can easily be configured for a new game and we can watch its 'behaviour'. Then focus on improving the search / eval / pruning etc. which can be tedious but you'll learn a lot. Btw. one issue i often encounter : an engine (even very strong ones) may 'think' a rather long time when only one move is possible in the position .. it seems the move generation method causes this, trying to be fast in most cases, but to a human this is weird - maybe you'll find a way to avoid such 'behaviour' ..

[tissatussa]

here's a nice game against that engine i once tried to create, called 'ARBE', written in Nim (kind of Python and can be compiled), just to learn. ARBE always plays the same variation at a certain max depth and it seems Catto has the same 'behaviour' (not sure though). I also posted this video on my BitChute channel, read the description there.

https://github.com/user-attachments/assets/e5b15b3d-e01a-46d5-82b2-bff7d5a8f4d1

[nguyenphuminh]

Interesting, this is also the one you mentioned above right?

[nguyenphuminh]

Also, I have just dropped v0.7 which should improve the performance. It also comes with time control, and moves are searched in each depth incrementally until stopped or timeout. The "position fen" error is also patched.

I will continue developing based on this list of todos: https://github.com/nguyenphuminh/Catto/projects?query=is%3Aopen

[tissatussa]

Interesting, this is also the one you mentioned above right?

Yes, and that video is short and sweet .. you didn't yet tell me you're a chess player yourself ? Or mainly a programmer ?

[tissatussa]

..I have just dropped v0.7..

Great! I will try it today, and play it !

[nguyenphuminh]

Yes, and that video is short and sweet .. you didn't yet tell me you're a chess player yourself ? Or mainly a programmer ?

Oh I'm mainly a programmer, not that good of a chess player lol

Great! I will try it today, and play it !

Nice!

[tissatussa]

..I will try it today..

compiling went well and it played its first game .. looks good !!

[tissatussa]

and i like to bring this to your attention : often Catto develops the 2 kNights quickly, without any (center) pawns .. i know many simple engines have this "style" but it's an inferior setup : the kNights can easily be chased by pawns and they often end on bad squares .. you should avoid Catto to play like this, better just start with 1.d4 or 1.e4 and develop the kNights after that. Here are two example positions :

( i could have created a new Issue for this )

[nguyenphuminh]

Yeah true. I probably can hardcode a rule to prevent this.

[tissatussa]

..I probably can hardcode a rule to prevent this..

yes, i was thinking that way too : i always disable the opening book of an engine because it's not fair .. but starting the game with 1 hardcoded move can't be wrong, i'd understand and allow it. Probably the next opening moves will be more harmonious.

same for the c-pawn : it's better to first play c2-c4 (/c7-c5) and THEN play the kNight to c3 / c6, otherwise the piece harmony can get crippled. This is just a general rule, born from practice, but not always true ..

[tissatussa]

another thing (i have too many ideas, help!) is capturing : in chess capturing is optional and this is a great "force" : it's best to let pieces / pawns BE captured, then decide how to re-capture and multiple ways of re-capture are preferable because the opponent should reckon the optimal take-back .. regarding that, at a next move the optimal recapture can be different .. so, it's good to keep pieces on the board, make the opponents pieces 'bad' by restricting them in their movement and then PUSH (esp. pawns) to get "more space". That's the way i play chess and i'm not the only one .. besides that, capturing a piece is often forcing, and it's your turn again.

[nguyenphuminh]

Thanks for the suggestions! I will be checking them out when I wake up.

[tissatussa]

..when I wake up..

yes, and take it easy .. you're in Vietnam, right ? I'm in Holland, Europe .. time difference .. we're into this chess engine thing now, and i follow you. My suggestions may lead you to a sound code architecture to develop, while Catto is still young, that's why i write all this.

many chess engine programmers use some tuning application / script to develop a next version .. recently i discovered 'Patricia', see https://github.com/Adam-Kulju/Patricia .. the author mentions the "EAS tool" by Stefan Pohl, which was a great help .. i have no experience with such tools, but it's good to know they exist : why create such tuning script yourself ? Maybe your Catto version isn't ready for such tuning, but it's good to know :-)

[nguyenphuminh]

another thing (i have too many ideas, help!) is capturing : in chess capturing is optional and this is a great "force" : it's best to let pieces / pawns BE captured, then decide how to re-capture and multiple ways of re-capture are preferable because the opponent should reckon the optimal take-back .. regarding that, at a next move the optimal recapture can be different .. so, it's good to keep pieces on the board, make the opponents pieces 'bad' by restricting them in their movement and then PUSH (esp. pawns) to get "more space". That's the way i play chess and i'm not the only one .. besides that, capturing a piece is often forcing, and it's your turn again.

I think I will have to think more about this.

[nguyenphuminh]

..when I wake up..

yes, and take it easy .. you're in Vietnam, right ? I'm in Holland, Europe .. time difference .. we're into this chess engine thing now, and i follow you. My suggestions may lead you to a sound code architecture to develop, while Catto is still young, that's why i write all this.

many chess engine programmers use some tuning application / script to develop a next version .. recently i discovered 'Patricia', see https://github.com/Adam-Kulju/Patricia .. the author mentions the "EAS tool" by Stefan Pohl, which was a great help .. i have no experience with such tools, but it's good to know they exist : why create such tuning script yourself ? Maybe your Catto version isn't ready for such tuning, but it's good to know :-)

I do not have much experience in tuning the evaluation function, and there are many ways to do this I think. Some generates a proper PST through tuning, some trains a whole nnue for eval. But from what I know they still require a good deterministic evaluation function before that for training. So when I improve my eval function enough, I think I will use it for tuning/training a nnue-based eval function.

[tissatussa]

..I think I will have to think more about this..

yes, let it sink in .. i have no clue how to implement such 'knowledge' into values, but i just wanted to let you know, because you're not a 'real' chess player like me.