NoobBook should be advisor ?

[tissatussa]

while testing NoobBook in Weiss v1.5-dev, i'm exploring https://www.chessdb.cn/queryc_en/ for the first time .. for me, the following game (see attached ZIP) is a great example : it gave me some questions for improvement ..

i set no NoobBookLimit (=0) .. this position arose :

shashchess-vs-weiss.zip

[Event "engine-vs-engine"]
[Site " [https://lichess.org/jdvW1yDS](https://lichess.org/jdvW1yDS) "]
[Date "2021.07.15"]
[White "ShashChess v17.1 no NN"]
[Black "Weiss v1.5 dev 210715a"]
[Result "1-0"]
[FEN "rnbqkbnr/pppppppp/8/8/8/4P3/PPPP1PPP/RNBQKBNR b KQkq - 0 1"]
[GameDuration "00:30:13"]
[PlyCount "133"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "720+5"]

 1.   e3 {Forced move.}      c5 {0.77s}
 2.  Nf3 {+0.07/30 19s}      d5 {0.52s}
 3.   d4 {+0.14/29 10s}      e6 {0.52s}
 4.   c4 {+0.14/29 19s}      cxd4 {0.52s}
 5. exd4 {+0.30/28 11s}      Bb4+ {0.52s}
 6.  Nc3 {+0.22/28 14s}      Nf6 {0.52s}
 7. cxd5 {+0.14/28 11s}      Nxd5 {0.52s}
 8.  Bd2 {+0.23/29 22s}      Nf6 {0.54s}
 9.   a3 {+0.29/32 54s}      Be7 {0.52s}
10.  Be2 {+0.30/30 22s}      O-O {0.52s}
11.  O-O {+0.30/31 45s}      b6 {0.51s}
12.  Ne5 {+0.61/30 16s}      Qxd4 {-0.51/31 31s}
13.  Bf3 {+0.76/30 11s}      Qxe5 {-1.07/30 26s}
14.  Re1 {+0.92/30 0.001s}   Qf5 {-0.96/29 42s}
15. Bxa8 {+0.84/29 17s} ...

position after 11.O-O : Black to move.

it's a game SF ShashChess v17.1 (without NNUE) vs. Weiss v1.5-dev. -- 12 min each side + 5 sec bonus.

here are the first few moves .. after 11.O-O it's Blacks turn : Weiss plays 11...b6 and gets a stunning reply : 12.Ne5 ! Weiss took pawn d4, which seems poison .. this position is just after the last existing NoobBook position : now, Weiss calculates on its own (you can see the CuteChess eval duration times in the PGN, 0.51 or so must be that connection speed with CDB) .. i checked the position online at https://www.chessdb.cn/queryc_en/ and move 11...b6 seems the only "known" move in this NoobBook, but maybe the pawn IS poison ..

i did some further tests with this position in SCID, with MPV 20, testing NN engines and non-NN : almost all NN engines drop the move b6 rather quickly, at depth 11 or so, but most non-NN engines favor b6 for a long time as one of the top moves !? We could consider this position a benchmark :-) You should study this one ..

All this brings me to this question : should we treat NoobBook like an Advisor ? : a source with its own move-evaluation-ranking-list of any position, but this list should only be "consulted" by Weiss, next to its own calculations / evaluations, then decide which "best move" is concluded.

i'm thinking of those (GitHub) projects which play chess according to 2 or more engines, often called master and slave(s). but i don't know how these programmers write their decision functions ..

what do you think about all this ?

NOTE: it's a pitty Weiss has no MPV - that way, we can not see the b6 ranking, and how it changes by rising depth ..

[TerjeKir]

Some info on noobbook: the only info weiss gets is one of the 'good' moves, i.e. moves with rank 2 in the table you see on the website. Sometimes these are bad, which is unfortunate, but as weiss queries the position (and further positions as weiss plays on) the database is set to analyse the positions again/better, and the error is likely fixed. Humans can also manually explore positions on the website to help improve the database. Due to this, noobbook will for example never give b6 in that position again, it now has Nc6 as the only rank 2 move in the position:

Given that the only info we get is a single move, it might be hard to combine with own analysis. As most tournaments and rating lists disallow using own books, noobbook is more of a toy than a big focus, but I will give it some thought :)

I will try to add multi-pv soon, I finally got around to trying the Nibbler GUI and multi-pv would be great for it!

[tissatussa]

Some info on noobbook: the only info weiss gets is one of the 'good' moves, i.e. moves with rank 2 in the table you see on the website. Sometimes these are bad, which is unfortunate, but as weiss queries the position (and further positions as weiss plays on) the database is set to analyse the positions again/better, and the error is likely fixed. Humans can also manually explore positions on the website to help improve the database. Due to this, noobbook will for example never give b6 in that position again, it now has Nc6 as the only rank 2 move in the position:

yes, i read the description page .. it's a cool idea !

Given that the only info we get is a single move, it might be hard to combine with own analysis.

why a single move ? Does the API not allow to query a list, like your screenshot ? if you have 2 MPV lists, and set a master / slave factor for each list, a simple addition would give a result MPV list, and top move is best. Btw. in this game i noticed Weiss often plays the worst ranked NoobBook move of a "2" subset !? Is this coincidence ?

As most tournaments and rating lists disallow using own books, noobbook is more of a toy than a big focus, but I will give it some thought :)

yes, there might be something in it .. when implementing it, we should make it great and simple .. maybe even have a few UCI options : how to perform / query the CDB. [..]

I will try to add multi-pv soon, I finally got around to trying the Nibbler GUI and multi-pv would be great for it!

it's really useful for (my) analysing, and using CuteChess demands MPV 1 anyway .. i once communicated with the creator of the Halogen engine, to implement this also .. and he did .. watch out for this error : when some time limit is set, and the evaluation loop ends, take the top move of the LAST fully calculated MPV list, not the current depth : doing so will often result in a worse move ..

[TerjeKir]

It's possible to query a full list, but parsing this and using the info in a useful way quickly becomes more complex than the feature is worth, as it's hardly ever allowed to be used. I might consider doing this at some point, but it's not a high priority for now I think.

The move returned by the database is chosen randomly from among the Rank 2 moves, so playing the worst one a lot is a coincidence (as far as I know).

Yeah, multi-pv will be very nice to have!

[tissatussa]

I agree on all you say .. i once tried Nibble GUI, now i remember .. i'm looking forward to your MPV mechanism .. do you ever contact other programmers to help you build / use new functions ? I guess your engine will already have some top-N list ..

[tissatussa]

regarding MPV i may say one more thing (or i could start a new issue) : in SCID i encounter nasty sorting errors in the MPV listings of SOME engines, not all .. that's very annoying : the eval fluctuation of all MPV moves is not clear while calculation continues and depth increases .. i know some puzzles which are hard to find for some engines but then it helps setting a high MPV : then the hidden best move indeed is considered and found to be very best at much higher depth .. do tournaments allow MPV ? It can result in better moves, but it slows down the program i guess ..

[TerjeKir]

I talk to devs of other engines daily, discussing possible improvements and such - Weiss has gained a lot from ideas shared in the community!

Weiss currently has no list of searched moves, it only remembers the best move of the last finished depth. So constructing move lists with their scores, and combining with a movelist from NoobBook would be quite a bit of work.

Multi-pv is allowed in tournament, but generally leads to weaker engine from what I've heard. It is very cool for analysis, and as you note sometimes allows the engine to find tricky moves much sooner. I have little experience using SCID, but if you encounter such problems with Weiss multi-pv when it is done I will try to fix it :)

[tissatussa]

you say it only remembers the best move of the last finished depth .. how can i understand this ? When running Weiss on some position in SCID (or any other GUI), with 1 MPV, i often see some move being top move for a short time and become best move again at higher depth !? How should i understand this ? So, some other moves ARE considered on the way ? I know a term Pruning exists, to determine which variations to cut (am i right?), but i'm not into that ..

[tissatussa]

you also say So constructing move lists with their scores, and combining with a movelist from NoobBook would be quite a bit of work .. i see some simple concept :

list 1     list 2
-----------------
 a3         c3
Nc2         a3
Kb5        Nd2
 c3        Kb5
Nd2        Nc2

in list 1 (eg. from CDB) has a3 best move but in list 2 the move a3 is second best .. we could give 5 to the top move, 4 points to the second move etc. so a3 will have 9, c3 has 7, Nc2 has 5, Kb5 has 5 and Nd2 will have 4 points : a3 is best move .. and, as i stated, you can give a factor to the values of a list, eg. 0.8 for the values of a CDB list, because its influence towards the final "best move" decision is considered less "reliable" !?

[TerjeKir]

Querying all (known) moves would give a list of all moves with their rank, score etc. Then I'd need to parse that and filter out everything not rank 2, as other ones are much less reliable.

For Weiss, all root moves are considered during the search, but after a depth finishes the only thing saved is the best PV and score - the score is then the score of the best root move. The other root moves and their individual scores are not saved anywhere, so I'd need to make a list of these with their scores during search and sort it.

After this I'd have the 2 lists which will often be 1-3 moves from noobbook and the full list from weiss. Now adding the lists should be fairly easy, but finding the right weight for each might not be.

While the concept is fairly simple, this is a fair bit of work and a lot of extra code (in C) for a toy that is used very little. If I find another reason to collect information about each root move (I know SF does it so there probably is something more generally useful to also do with it) I can revisit this idea as the additional code specifically for this would then be much less :)

[tissatussa]

thanks for taking the time to contemplate & explain your knowledge / ideas / insights in this matter .. i do not want to force you implementing anything, i just share my ideas .. i've no experience in programming a chess engine in C and i can imagine the work it takes to build such functions or figure out / discuss how to use existing modules .. your detailed info can be helpfull to anyone reading this issue :-)

i think it's clear now, i have nothing to add .. i guess you can close this issue now .. however, i might add some info here in the near future. I learned from other GitHub programmers that writing issues like this should stick to the subject only, so discussing eg. MPV can better be done in a new issue.

same for some special (puzzle) positions : regularly i test intriging positions which i find, and i can report some results in a new issue .. are you interested ? Can such positions be of any help for you to improve the way Weiss does pruning / searching ?

[tissatussa]

you could take a closer look at the latest Rodent IV engine (v0.32) .. this project has now ended, the creator decided he no longer has time to develop it .. Rodent is mainly focused on 'personalities', not (!) playing strength : each personality has different parameters and the user can tweak those .. especially the Bosboom personality (an extarordinary Dutch player) is interesting.

you'll find this text part in my recent Halogen issue "develop engine with well chosen positions ?" https://github.com/KierenP/Halogen/issues/239

what do you think about those Personalities ? Many engines have some parameters regarding the piece values, and other factors. However, i have no clue how to test while tuning them.

[TerjeKir]

Feel free to make as many issues as you'd like, I'm grateful for all inputs. Already 2 of your ideas implemented that I'm happy to have, and multi-pv will come soon :)

I enjoy testing Weiss with difficult positions as well (though many in my usual set that used to be hard, or even impossible, for Weiss are now too easy :D), and I'd be interested to see any cool positions you have that Weiss does poorly (or well!) in!

I tried the positions from the Halogen issue, and Weiss finds Qxg6 best after a bit over a minute (but low score), fails to find Qf6+ (this one is very hard!), finds Qd3 in under a minute (with huge score), and Bc4 in a couple seconds (score starting low but rising). A pretty good result I think!

I use 1 thread, but 1 GB hash so might be slower with 32MB hash :)

As for personalities, I have no plans for that in the near future, but could be cool at some point in the future!

[tissatussa]

you say ...[positions] that used to be hard, or even impossible, for Weiss are now too easy... how can i understand this ? is it positions LIKE that or exactly THAT position ? Or are any "patterns" involved to find "hidden best moves" ? I will submit more test positions in another issue.

using hash and its advantages is unclear to me .. in the Halogen issue i mention :

I still have no clue how to use Hash .. engines have (very) different default values for it, maybe because their architecture differs that much .. I read that setting Hash to high is not good practice, because using much Hash can slow down the evaluation instead of helping it. The Readme text of some engine even pointed out a method for the user to determine an optimal Hash value for their PC, but I don't remember which engine it was, sorry .. I used that procedure (for that engine) but the results were unclear to me .. I think 4 GB is way to much, but I can be wrong, not being a chess engine programmer myself.

so, i assume a big hash can slow down the overall speed, also with MPV, i often see a default UCI option of 32 Mb or so, not Gb ..

as for personalities : what do you know about a way to tune such parameters anyhow ? Are test positions being used, which have a known "best move" (like a puzzle) ? Or is this just another "toy" ?

[tissatussa]

i just found this link : https://github.com/feldi/py-goratschin it also mentions an "Advisor" and gives a link to that module .. this is the python script i once tried, but was hard / unclear how to test the results / any differences ..

[TerjeKir]

For the comment about positions becoming easy, that's just about a set of positions I have collected to test Weiss since a long time ago. Some of the positions it used to struggle with a year ago are now trivially easy :)

The hash is used to remember the result of searching positions encountered throughout the search, to avoid searching them again, so the bigger the hash, the more positions the engine can remember. The optimal hash size will depend on how long you intend to search for; If you're having engines play very quick matches, then big hash isn't useful as they don't search enough positions to fill it anyway, and the size can slow it down a bit. However, for analysis going for minutes (or using multiple threads) a bigger hash will be beneficial.

As an example, searching your position with Qxg6 as best move and 1GB hash, after 1 minute Weiss has made use of 50%, 2 minutes 65%, 5 minutes more than 90%. This will vary from engine to engine, but for Weiss I think 1GB is reasonable for analysis of 1-5 minutes. With more threads the time to use up the hash will shorten significantly, so adjust accordingly :)

In general a hash big enough that it doesn't fill too fast is best, but it can also happen (rarely?) that the engine forgetting a search result and being forced to redo it, will let it discover something it missed the first time, despite searching to the same or lower depth (due to different positions in the hash, different history scores etc since the last time it searched it), so there will be some cases where an engine finds a trick faster with small hash.

[TerjeKir]

As for personalities, I think there are generally 2 broad types of personalities:

• The first is personalities designed for different playstyles, often to give human opponents more variation and enjoyment out of playing against the engine. Used together with skill-reducing measures (or just a weak engine to begin with) to allow the humans to have a chance at winning.

• The second is tactical personalities, designed to be better at finding tricky moves like sacrifices (though at a cost of general playing strength). Crystal is in a way such a personality for Stockfish, and Houdini's tactical mode was another example.

I'm not interested in the first; Weiss is not a good opponent for humans, and I have no intention of trying to change that. The second one is interesting to me, and I would like to explore this at some point :)

[TerjeKir]

Multi-PV is now added :)

[tissatussa]

wow !! i will try it .. did you consider / follow my tips about MPV errors when quiting the evaluation ? I read Implementation inspired by Ethereal .. so you used some existing functions and implemented them ? Btw. several Ethereal versions have many small errors / flaws while using CuteChess .. also its MPV list sorting in SCID is chaotic .. anyhow, i will let you know !

[tissatussa]

your MPV seems to work .. that is, when i use this new Weiss version in my own GUI .. but NOT in SCID ? It still shows only 1 line .. you should certainly take a look at this .. SCID is easy to install - i think even for Windows, but i'm not sure .. do you use Linux, like me ?

[TerjeKir]

I implemented it in mostly the same way Ethereal did. Unlike Ethereal I turned off printing fail high/lows (when the info print has 'score cp xxx upperbound/lowerbound') when multipv is on, as it was way too chaotic with that many prints in cutechess.

I believe I have now fixed the issue SCID had, let me know if it works for you!

[tissatussa]

indeed the "MPV listing" in SCID is now much better : i see not 1 line but all MPV lines when MPV more then 1 .. but the sorting is not OK, same as Ethereal (as i mentioned) .. see this screenshot of the position of this issue after 11...b6 (White to move) :

you see the order is not right : the move Ne5 has eval +0.54 and this is the highest eval but it's not on top .. most other engines do NOT suffer from this .. i have no clue what's causing this .. but it's annoying for the user : the fluctuating list is not properly displaying the progress ..

[TerjeKir]

I see, this happened here too. I will look into a fix, but I can't promise anything as it might require a lot of extra code.

[TerjeKir]

I managed to change it so it now sorts the multipv lines by score. It will still only update the lines once all lines at that depth are done, unlike SF which prints an update whenever it finishes a line, so it might not be perfect still. I hope it's better than before, let me know :)

[tissatussa]

i tested that newest version and it seems OK : the lines are now sorted properly ! Indeed all MPV lines of a next depth are printed as soon as they are all calculated .. this is simple and clear, i like it (and Weiss is not the only engine performing like this). We might conclude this functionality is even better then the "dynamic fluctuating" sorted lines which SF and others show .. keep it like this !

[TerjeKir]

Thanks for the feedback! I will keep it as is then :)

[tissatussa]

here, some of our subjects are out of my "NoobBook should be advisor" scope .. if i want to communicate with you without starting a new "issue", can i reach you by email ? I prefer that, despite Discord etc

[TerjeKir]

I don't check my email very often so I'd recommend getting on discord!

I also don't mind having more issues here as long as the topics are relevant to Weiss :)

[tissatussa]

OK, i heard about Discord, some people i know gather in Discord group also .. so, i will try .. what is the URL and your handle ?