Chess board to FEN working correctly on DEMO, but not locally

TomasTillmann commented 1 year ago

Hi,

not sure what I'm doing wrong, for this image: whole-board , the website demo returnes correct fen string, but local installation doesnt' work.

Interestingly, the example --url image works locally just fine.

pip list output:

absl-py              1.4.0
astor                0.8.1
beautifulsoup4       4.12.2
certifi              2023.7.22
charset-normalizer   3.2.0
gast                 0.2.2
google-pasta         0.2.0
grpcio               1.56.2
h5py                 2.10.0
idna                 3.4
importlib-metadata   6.7.0
Keras-Applications   1.0.8
Keras-Preprocessing  1.1.2
lxml                 4.9.3
Markdown             3.4.3
MarkupSafe           2.1.3
numpy                1.18.5
opt-einsum           3.3.0
Pillow               5.4.1
pip                  10.0.1
protobuf             3.20.3
requests             2.31.0
setuptools           68.0.0
six                  1.16.0
soupsieve            2.4.1
tensorboard          1.15.0
tensorflow           1.15.5
tensorflow-estimator 1.15.1
termcolor            2.3.0
typing-extensions    4.7.1
urllib3              1.26.6
Werkzeug             2.2.3
wheel                0.41.0
wrapt                1.15.0
zipp                 3.15.0

python --version output:

Python 3.7.0

Executing by: python .\tensorflow_chessbot.py --filepath "C:\Users\tomas\Pictures\Screenshots\whole-board.png" Output:


--- Prediction on file C:\Users\tomas\Pictures\Screenshots\whole-board.png ---
         Loading model 'saved_models/frozen_graph.pb'
         Model restored.
Closing session.
Per-tile certainty:
[[0.982 1.    0.977 1.    0.749 1.    0.874 1.   ]
 [1.    1.    1.    1.    1.    1.    1.    1.   ]
 [1.    0.999 1.    0.998 1.    0.999 1.    1.   ]
 [1.    1.    0.817 1.    0.987 1.    0.999 1.   ]
 [0.999 1.    0.998 1.    0.641 1.    0.999 1.   ]
 [1.    0.989 0.82  0.996 0.921 0.802 0.435 0.995]
 [0.949 0.881 1.    1.    1.    0.998 0.966 0.999]
 [0.988 1.    0.584 1.    0.747 1.    0.979 1.   ]]
Certainty range [0.435048 - 1], Avg: 0.954174
---
Predicted FEN:
8/8/2P1P3/2N1N3/2r1bb2/8/PP4R1/2K3p1 w - - 0 1
Final Certainty: 43.5%

Thanks.

TomasTillmann commented 1 year ago

Just tried the example_input.png image, and that works as well, with 100% accuracy.

Elucidation commented 1 year ago

Likely the JavaScript online model and the local Python one have some preprocessing step differences such as image resize etc. , Have a look in https://github.com/Elucidation/ChessboardFenTensorflowJs

I don't remember if the model used between the two of those is the same or different, that's possibly another reason.

If you don't find anything let me know

On Sat, Jul 22, 2023, 06:09 Tomas Tillmann @.***> wrote:

Just tried the example_input.png image, and that works as well, with 100% accuracy.

— Reply to this email directly, view it on GitHub https://github.com/Elucidation/tensorflow_chessbot/issues/41#issuecomment-1646580963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADKF3PBIZMOIW46DFAING3XRPGG7ANCNFSM6AAAAAA2T2LIWQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

TomasTillmann commented 1 year ago

I think it's perhaps about the dimensions of the input picture. Does it have to be on some specified dimensions? It seems to me like the web version scales it, and this one doesn't.

Elucidation commented 1 year ago

There's no specific preferred size, it eventually gets scaled down to something like 32x32 pixels per chessboard tile, so possibly what's going on is the lined pattern texture from the image you shared is confusing the Python model whereas the JavaScript web one has scaled it down to where it's blurry enough to not interfere. You can test this by manually trying different resizes and see what works for you

On Sat, Jul 22, 2023, 11:54 Tomas Tillmann @.***> wrote:

I think it's perhaps about the dimensions of the input picture. Does it have to be on some specified dimensions? It seems to me like the web version scales it, and this one doesn't.

— Reply to this email directly, view it on GitHub https://github.com/Elucidation/tensorflow_chessbot/issues/41#issuecomment-1646650243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADKF3OT2UD3JL25GVUYLSLXRQOX5ANCNFSM6AAAAAA2T2LIWQ . You are receiving this because you commented.Message ID: @.***>

TomasTillmann commented 1 year ago

Yes, that seems to be the reason. Thanks.

Do you think it could be possible to do something about it though? I experimted a bit, and if I blur the image, making the quality of the image lower, it works correctly. But that feels like a hack ..., and it also makes the work for the model harder, since the pieces are not as sharp as they could be. Potentially leading to wrong piece prediction.

I think it would be very nice for the chessboard recognizer to understand even if the black tiles are hatched and not fully black colored, where the grid lines are. Do you think it could be done? Perhaps by feeding it boards with hatched black tiles?

I feel like majority of chess pdf books are having black tiles hatched, so it could be a really useful improvement. To name some: The woodpecker method, Encyclopedia of chess tactics, ...

Elucidation commented 1 year ago

Good to hear, and yes there are 2 parts,

1 is that the model isn't really trained on those type of images so it could definitely benefit from further training on those sorts of labeled images.
2 is that likely the issue is more on the computer vision chessboard detection side, not the ML piece recognition side, where the chessboard detection routine essentially looks for certain X and Y gradient patterns, and the hatched pattern is disrupting that at full resolution, but those gradients go away when blurred. The 2nd part can be tackled by modifying the chessboard detection routine, either naively by blurring/reducing high frequency gradients like the hatch pattern, or something more sophisticated (probably unnecessary). And then once the board and tiles have been detected, use pass the the original image resolution tiles to the ML model. I'll open an issue for this

On Tue, Jul 25, 2023 at 9:30 AM Tomas Tillmann @.***> wrote:

Yes, that seems to be the reason. Thanks.

Do you think it could be possible to do something about it though? I experimted a bit, and if I blur the image, making the quality of the image lower, it works correctly. But that feels like a hack ..., and it also makes the work for the model harder, since the pieces are not as sharp as they could be. Potentially leading to wrong piece prediction.

I think it would be very nice for the chessboard recognizer to understand even if the black tiles are hatched and not fully black colored, where the grid lines are. Do you think it could be done? Perhaps by feeding it boards with hatched black tiles?

I feel like majority of chess pdf books are having black tiles hatched, so it could be a really useful improvement. To name some: The woodpecker method, Encyclopedia of chess tactics, ...

— Reply to this email directly, view it on GitHub https://github.com/Elucidation/tensorflow_chessbot/issues/41#issuecomment-1650163565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADKF3ILYYFGFRWNGNJHIMDXR7YBFANCNFSM6AAAAAA2T2LIWQ . You are receiving this because you commented.Message ID: @.***>

Elucidation / tensorflow_chessbot

Chess board to FEN working correctly on DEMO, but not locally #41