riels89
commented
1 year ago This is resolved. It turns out the train data was not fully downloaded. I downloaded it from the google drive link, so if anyone else is having this issue double check that the download from drive was actually correct.
When running the train.ipynb for NMNIST I am unable to reproduce the test accuracy. I only reach sub 30% accuracy rates. The model is clearly overfitting based on the fact that I reach higher train accuracies faster than the existing numbers. The first epoch I get 90% train vs 80% expected. Similar story when it comes to train loss loss, which is lower significantly faster 0.26 after first epoch compared to 0.4 expected. Test loss is always much higher, 1.0 compared to <0.2. To be clear I trained the models far longer than a single epoch, this just better shows the discrepancy I am seeing. Messing about with hyper parameters also gave no success.
I tried loading the existing trained model in nmnist/Trained and this gave the expected test accuracy numbers. When further training the pretrained model the accuracy degrades quickly.
I downloaded the NMNIST data manually and the only change I made was to an assert statement in the NMNISTDataset because it was incorrect:
assert len(glob.glob(f'{data_path}/')) == 0->assert len(glob.glob(f'{data_path}/')) != 0To reproduce current behavior Run train.ipynb.
Expected behavior Expected to get accuracy values in the high 90s as shown in the notebook and on the website.
Environment (please complete the following information):