search

microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.51k stars 4.28k forks source link

MNIST sample model poor classification #1125

Closed codec-abc closed 7 years ago

codec-abc commented 7 years ago

Hi,

I tried to test the different sample models for the MNIST data set using my own data (by modifying the EvalClient project)

It run fine but the result seems wrong wrong way too often. As an example using the following data with the trained 03_OneConvdropout model:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 36 124 254 255 120 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 48 158 255 255 255 255 166 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 48 255 255 255 255 160 118 26 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 38 246 255 255 152 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 6 210 255 255 60 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 130 255 255 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 4 255 255 212 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 64 255 255 72 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 138 255 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 164 255 204 0 0 22 58 58 28 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 208 255 154 94 208 255 255 255 255 188 62 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 244 255 255 255 255 255 255 255 255 255 255 66 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 232 255 255 255 230 106 58 58 74 230 255 255 36 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 210 255 255 206 0 0 0 0 0 46 255 255 138 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 236 255 180 0 0 0 0 0 0 0 150 255 255 14 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 162 255 220 0 0 0 0 0 0 0 80 255 255 42 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 92 255 255 112 0 0 0 0 0 0 76 255 255 24 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 8 236 255 255 156 4 0 0 0 22 168 255 182 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 54 255 255 255 246 142 134 146 250 255 255 74 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 58 222 255 255 255 255 255 255 255 112 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 44 114 168 218 192 116 34 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I got the following result: -2.277135 -7.73908 -1.059328 -2.477858 -6.769311 7.046165 4.724845 -9.132572 10.03633 3.570117

While the data seems to represent a "fine" six digit it has a pretty low score. On average I would say that for digit that "look correct" I have an error rate of 10% to 50 % (depending on which digit) which is way more than the most basic model.

Is this something expected or am I doing something wrong (which clearly could be since I am little to none experience in machine learning)?

SivilTaram commented 7 years ago

What the error rate the CNTK tool tell you when testing after the training? If the error rate is high, I guess that your data is not poured into CNTK using correct reader format. Could you show some more detailed information?

codec-abc commented 7 years ago

Sorry, I ditn't explained myself clearly. I trained the various models using the documentation and it worked fine. The error rate was exactly those stated. Then I made a paint like application where the drawing can be send to the model. The problem is that the prediction were not so good.

I made more tests and it seems that using a the ConvNet_MNIST helps. Moreover, I noticed that it had difficulties if the digit drawn was a bit too small or the value were not saturated enough. I expected the network to be more robust for those cases.

cha-zhang commented 7 years ago

03_OneConvdropout is a model in GettingStarted and it's deliberately kept extremely simple. As a result the classification accuracy is bad. In our examples the best model for MNIST would be ConvNet_MNIST.py, which is state-of-the-art with no data augmentation.

If you want to further improve accuracy, to handle your own data, the best way is to add your own training data. Another way is to use data augmentation. For example: https://www.microsoft.com/en-us/research/publication/best-practices-for-convolutional-neural-networks-applied-to-visual-document-analysis/.

codec-abc commented 7 years ago

Thanks for the links. I can confirm that using the ConvNet model helps. Moreover the other problem that I had (and probably even worse than using a too simple model) is that I drawn on a bigger texture than the model use as input. When I resized it I lost too much information making the prediction hard to do. Anyways, thanks for your help and sorry for the inconvenience.