reidmcy
commented
3 years ago Hi Vitaly,
What you're seeing is one of the limitations neural networks. The models rely on the training data from lots of previous games. They can be thought of as trying to find the average of the moves they've seen. Which means if the board is very far away from what they're trained on they behave weirdly. And to get into that situation requires both players to be messing around, so it's likely the nearest game is not a good example of play. So yes, we'd have to retrain them to not have this, but getting all the possible weird endgames is nearly impossible.
Other chess engines get around this issue by using search, so looking at possible moves and selecting the ones that eventually lead to wins. An endgame table (or tablebase) would also help solve the issue, but we don't use them for the bots. We are looking into ways of predicting when humans resign so are hoping having a realistic resignation will help. Also, adding some 5 fold repetition avoidance.
P.S. It's great to hear the dataset is useful
Hi. Sorry for bringing up the quality of the moves again; this one seems like not a single fluke, but as as systematic deficiency: Maia 1100 fails to win a game with 6 queens vs. a king. The game is here (also attached). On move 66 Maia creates its sixth queen by promotion, and then for the next 47 moves it only shuffles its pieces around on the 1st rank, without even trying to attack the king. The game finally ends on move 113 by the 5-fold repetition as a draw.
It seems that in this case the net optimizes for moving the pieces to the 1st rank, rather than winning the game. Maybe aggressive king movement is the way to beat Maya? :)
(Note that in this game the first few moves Maia makes are actually from an opening book; I've extracted it from your fairly useful dataset).
Anyway, I understand that fixing behavior in such cases would require re-training the net (or at least training it some more), which I don't know if you want to do. Still, in a 6-queen end game, at least moving the queens to attack is something that players of any rating would do.
So I guess I'm looking for an advice here. I'm trying to use Maia as a beginner-friendly AI; is there anything that I can do to make it actually win in this example? Maybe use end-game tables of some sort? Do you do that for the lichess bots? Or supplement it with a different AI that would recognize obvious moves, and take over under some conditions? Any ideas? (I would of course prefer Maia to just work by itself).