Reproducing paper results on CIFAR-10

[rsundar96]

Hi,

I've been executing your code using the hyper parameters detailed in _main_reproducepaper.py for the VGG19 and ResNet50 models on CIFAR-10 in order to reproduce the results detailed in your paper. More specifically, I am trying to reproduce the results of Figure 4(c). However, I do not seem to be getting a Test Accuracy greater than 82% for the VGG19 model (Curriculum and Random initialisation) on CIFAR-10.

I first tried the following set of hyperparameters:

• Optimizer : SGD
• Batch Size : 100
• Epochs : 140
• Learning Rate : 0.12
• Learning Rate Decay : 1.1
• Minimal Learning Rate : 1e-3
• Learning Rate Batch Size : 700
• Order : vgg19

This did not result in the same values depicted in Figure 4(c). However, upon using the following set of hyper parameters:

• Optimizer : SGD
• Batch Size : 100
• Epochs : 300
• Learning Rate : 0.1
• Learning Rate Decay : 1.5
• Minimal Learning Rate : 1e-3
• Learning Rate Batch Size : 17,500
• Order : vgg19

I obtained a Training Accuracy of 91% and Test Accuracy of 81.2% (Curriculum Learning) and a Training Accuracy of 88% and Test Accuracy of 81.3% (Random Initialisation).

Would it be possible for you to share the hyper-parameters that enabled you to obtain Test Accuracies of >90% in your paper? A VGG19 model should be able to attain an accuracy ~92% on the Test Set. Please do let me know in case I have done something incorrectly on my part.

[gust0717]

Hi there. Did you have any problem reproducing the paper? I couldn't manage to run the functions that reproduce the experimental cases.

[rsundar96]

No, I was unable to reproduce the results shown in Figure 4(c). I got in touch with Hacochen regarding this and he said that the VGG and ResNet code have not been incorporated into the repository. Whatever results I have mentioned in my previous comment were the best that I could achieve on CIFAR-10.

[gust0717]

I see. But were you able to reproduce the main experimental cases (especially case 1) without any problems? I've just copied the code but got this error upon running curriculum_small_mammals:

I supposed the code was all ready for running, but I just can't seem to make it work. Any ideas on what the problem may be?

[vinusankars]

Hi, the code available is to reproduce Fig. 4b in the paper. It uses a moderate sized CNN architecture with 8 convolution layers and inception based scores. Fig. 4c uses the inception scores to train a VGG-based network (and that code is not provided in the repo).

The repo is useful in reproducing Fig. 2, 3, and 4b.