Relatively good quality image fails in a bad way

[pythonflaskserverapps]

I admire your project very much. I created a tool with which you can drag-drop upload an image and recognize the FEN on it using your neural network ( https://github.com/pythonflaskserverapps/fenbot ).

I opened a thread about your project and later my tool on lichess: https://lichess.org/forum/general-chess-discussion/bot-that-recognizes-fen-on-arbitrary-screenshots-that-contain-a-chess-board.

In the thread a user posted this image which is relatively good quality but failed badly:

https://i.imgur.com/8yVD3g6.png

prediction: 1Q2Q3/8/p7/RPqq4/8/8/8/8 ( 45% certainty )

I wonder if you can give an explanation for what's going on here. Does the fact that the board is somewhat rotated play a role here?

[ddugovic]

Out of curiosity, do you (or the user) have a collection of multiple such images?

[pythonflaskserverapps]

I don't have such images.

I don't know if the user has more of such images.

[ddugovic]

I started digging into this code and it seems that it uses PIL instead of OpenCV. I guess I'll try rewriting parts of this using OpenCV instead and see if I can guess how to test that I haven't broken anything?

[Vinvin20]

Some images not well recognized (sometimes the black king become a queen black or white) :

Here some images with lower quality :

[Vinvin20]

https://i.imgur.com/8yVD3g6.png prediction: 1Q2Q3/8/p7/RPqq4/8/8/8/8 ( 45% certainty ) I wonder if you can give an explanation for what's going on here.

I suggest the reason is the "hatched squares". Very often black squares have a solid colour, but not in this black and white picture.

[ddugovic]

Also, chess boards can be monochrome: http://xkcd.imrannazar.com/content/img/3935.jpg

[Vinvin20]

Not so bad on this one : 24 pieces well recognized :

[Elucidation]

Hi @pythonflaskserverapps , thanks for the kind words and making that tool, that's great! Yes, the model was not trained using images such as that one with the striped background, so it's likely it's failing due to that.

@Vinvin20 that's a useful list of failure images. I would guess there are a two main causes of failures: 1 - computer vision detection fail/false positives, the board detection algorithm looks for strong alternating gradients vertically and horizontally to try and position the board. If the board has a thick black outline, or it is tilted, it will have a hard time finding the board or may find a bad version of it. https://github.com/Elucidation/tensorflow_chessbot/blob/master/tensorflow_compvision.ipynb is a bit old but explains the rough concepts.

2 - The image is a bit too different from what the model was trained on. In this case, some of the images have interesting edge effects like a white outline or low resolution, or high-frequency noise due to jpg compression, or a blue haze, such as in the examples you've shown. The (extremely basic) model likely just doesn't know what it's seeing as a result, or falsely thinks it's seeing something else instead, causing the bad predictions you're seeing.

@ddugovic switching from PIL / opencv shouldn't break anything, perhaps be on the lookout for RGB vs BGR difference, but the model will expect grayscale input either way.