Use "beautiful_mnist" from tinygrad

certik commented 7 months ago

This gets TF training work.

TODO:

[x] Extract the weights to GGUF
[ ] Get PyTorch inference working via GGUF weights
[x] Get NumPy inference working via GGUF weights
[x] If everything works, upload to HuggingFace
[ ] Get C inference working via GGUF weights and LLIR

certik commented 7 months ago

Results from full training:

$ python mnist-tf.py train mnist-cnn-beautiful-model
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 24, 24, 32)        832       

 conv2d_1 (Conv2D)           (None, 20, 20, 32)        25632     

 batch_normalization (Batch  (None, 20, 20, 32)        128       
 Normalization)                                                  

 max_pooling2d (MaxPooling2  (None, 10, 10, 32)        0         
 D)                                                              

 conv2d_2 (Conv2D)           (None, 8, 8, 64)          18496     

 conv2d_3 (Conv2D)           (None, 6, 6, 64)          36928     

 batch_normalization_1 (Bat  (None, 6, 6, 64)          256       
 chNormalization)                                                

 max_pooling2d_1 (MaxPoolin  (None, 3, 3, 64)          0         
 g2D)                                                            

 flatten (Flatten)           (None, 576)               0         

 dropout (Dropout)           (None, 576)               0         

 dense (Dense)               (None, 10)                5770      

=================================================================
Total params: 88042 (343.91 KB)
Trainable params: 87850 (343.16 KB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________
Epoch 1/15
422/422 [==============================] - 19s 45ms/step - loss: 0.2615 - accuracy: 0.9217 - val_loss: 0.1378 - val_accuracy: 0.9697
Epoch 2/15
422/422 [==============================] - 19s 46ms/step - loss: 0.0728 - accuracy: 0.9769 - val_loss: 0.0506 - val_accuracy: 0.9852
Epoch 3/15
422/422 [==============================] - 20s 47ms/step - loss: 0.0518 - accuracy: 0.9839 - val_loss: 0.0392 - val_accuracy: 0.9888
Epoch 4/15
422/422 [==============================] - 20s 48ms/step - loss: 0.0427 - accuracy: 0.9866 - val_loss: 0.0477 - val_accuracy: 0.9857
Epoch 5/15
422/422 [==============================] - 21s 50ms/step - loss: 0.0377 - accuracy: 0.9881 - val_loss: 0.0414 - val_accuracy: 0.9897
Epoch 6/15
422/422 [==============================] - 21s 50ms/step - loss: 0.0330 - accuracy: 0.9896 - val_loss: 0.0328 - val_accuracy: 0.9913
Epoch 7/15
422/422 [==============================] - 22s 51ms/step - loss: 0.0293 - accuracy: 0.9908 - val_loss: 0.0339 - val_accuracy: 0.9907
Epoch 8/15
422/422 [==============================] - 22s 52ms/step - loss: 0.0250 - accuracy: 0.9916 - val_loss: 0.0347 - val_accuracy: 0.9918
Epoch 9/15
422/422 [==============================] - 22s 52ms/step - loss: 0.0258 - accuracy: 0.9915 - val_loss: 0.0330 - val_accuracy: 0.9922
Epoch 10/15
422/422 [==============================] - 22s 53ms/step - loss: 0.0226 - accuracy: 0.9925 - val_loss: 0.0314 - val_accuracy: 0.9923
Epoch 11/15
422/422 [==============================] - 22s 52ms/step - loss: 0.0204 - accuracy: 0.9935 - val_loss: 0.0267 - val_accuracy: 0.9928
Epoch 12/15
422/422 [==============================] - 22s 53ms/step - loss: 0.0192 - accuracy: 0.9938 - val_loss: 0.0354 - val_accuracy: 0.9912
Epoch 13/15
422/422 [==============================] - 22s 53ms/step - loss: 0.0179 - accuracy: 0.9940 - val_loss: 0.0336 - val_accuracy: 0.9912
Epoch 14/15
422/422 [==============================] - 23s 54ms/step - loss: 0.0165 - accuracy: 0.9942 - val_loss: 0.0301 - val_accuracy: 0.9927
Epoch 15/15
422/422 [==============================] - 23s 54ms/step - loss: 0.0172 - accuracy: 0.9941 - val_loss: 0.0357 - val_accuracy: 0.9923
Test loss: 0.02838047407567501
Test accuracy: 0.9918000102043152
Keras model saved to 'mnist-cnn-beautiful-model'

certik commented 7 months ago

TODO:

[x] save the affine vector in batch norm to gguf
[x] ...and set it in pytorch
[ ] check layer by layer to see agreement

certik / mlc

Use "beautiful_mnist" from tinygrad #79