georgestagg / mnist-classifier-blas-wasm

MNIST Classifier - A WebAssembly BLAS Demo
https://georgestagg.github.io/mnist-classifier-blas-wasm/
MIT License
6 stars 2 forks source link

Obvious 1 strongly detected as a 7 #1

Open Hexstream opened 5 months ago

Hexstream commented 5 months ago

I'm not sure how actionable this is as a bug report, but I got this result here:

obvious-1-strongly-detected-as-7

Hexstream commented 5 months ago

Even worse:

brutally-unrecognized-7

Photosounder commented 1 month ago

It's literally impossible to get a 6 recognised as a 6. This is the clearest 6 I could draw, the others are even more clearly recognised as 5s. 2024-09-11 13 39 54

Hexstream commented 1 month ago

Here are my 5 attempts at a 6, I got 4 successes and 1 failure.

6-win-1.png

6-win-2.png

6-win-3.png

6-win-4.png

6-fail.png

georgestagg commented 1 month ago

This is really not intended to be a state of the art computer vision model, instead as a small demonstration of using Fortran BLAS routines under WebAssembly.

The original Multilayer Perceptron model proposal is from 1958. So, yes, the model is not great. Probably nothing can be done other than replacing the model with something better.

For me it seems to work best on drawings closer to the scaled digits in the original MNIST dataset, where the numbers are fairly small and centred in the box. Probably it's overfitting to that source data, but it does not matter because model performance isn't really the point of the demo. The code tries to scale and crop digits, but I suppose line thickness might make a big difference.

Screenshot 2024-09-11 at 12 57 59 Screenshot 2024-09-11 at 12 58 06

Saying that, I don't find it "literally impossible to get a 6 recognised". So maybe something is going wrong on your machine.

Screenshot 2024-09-11 at 12 53 41

If it really is that bad, useful information would be the browser and OS you're using so I can try to reproduce.

Photosounder commented 1 month ago

Saying that, I don't find it "literally impossible to get a 6 recognised". So maybe something is going wrong on your machine.

If it really is that bad, useful information would be the browser and OS you're using so I can try to reproduce.

I see what the problem is, you guys draw 6 with a small loop at the bottom, I draw them with a bigger loop. When I draw a smaller loop that doesn't take up half of the vertical space of the glyph it does detect a 6. I suspect that it recognises the loop reaching the vertical centre as a defining feature of 5, I guess 6 was trained on small loops.

Hexstream commented 1 month ago

Here are my 6 attempts at a big-looped 6, I got 3 successes and 3 failures, but the last one is a very near miss.

big-looped-6-win-1.png

big-looped-6-win-2.png

big-looped-6-win-3.png

big-looped-6-fail-1.png

big-looped-6-fail-2.png

big-looped-6-fail-3.png