search

imirzadeh / Teacher-Assistant-Knowledge-Distillation

Using Teacher Assistants to Improve Knowledge Distillation: https://arxiv.org/pdf/1902.03393.pdf
MIT License
256 stars 47 forks source link

Three main issues: seed optimization, parameter tuning on the test set, and wrong normalization for cifar10 #19

Open tknovisky opened 3 years ago

tknovisky commented 3 years ago

I have three concerns about this experiment:

  1. Seed optimization
    According to the paper and the code, it seems that the random seed is in the search space which implies seed optimization. A proper way is to repeat the experiment with different seeds and report the average result.

  2. Parameter tuning on the test set
    According to the current codes, the algorithm tunes hyper-parameters, e.g., seed, lambda, and temperature, on the testing set, which obviously violates the parameter tuning rule in machine learning.

  3. Wrong normalization for CIFAR10 In data_loader.py, authors normalize images by normalize = transforms.Normalize(mean=[0.507, 0.487, 0.441], std=[0.267, 0.256, 0.276]), which is wrong for CIFAR10, because these means and stds are computed based on CIFAR100. The correct means and stds are normalize = transforms.Normalize(mean=[0.49139968, 0.48215841, 0.44653091], std=[0.24703223, 0.24348513, 0.26158784]). Please refer to https://github.com/Armour/pytorch-nn-practice/blob/master/utils/meanstd.py for the correct way to compute these normalization constants for CIFAR10 and CIFAR100.

imirzadeh commented 3 years ago

Hi,

Thank you for your comments! Let me explain a few things:

1- Seed Optimization This is done not to optimize for better seeds and is done to have consistent and reproducible results. So to calculate the results, the results were calculated using best params (excluding seed).

2- Parameter Tuning on Test Set As you correctly mentioned, there will be a "leak" from the test set. To be honest, I wasn't aware of this leakage when performing experiments and I was following the literature, which unfortunately exploited this leakage. However, all baselines were trained in this manner and this might alleviate the situation a little bit. But you are right. But I believe this doesn't disclaim the effectiveness of the TAKD method. You can see another independent benchmark here: https://github.com/karanchahal/distiller

3- Normalization for CIFAR-10 You are right. These numbers seem to be different in practice. However, to the best of my knowledge, the input images CIFAR-10 and CIFAR-100 images are roughly the same. I don't see why they should be normalized differently. As you can see, the values you have provided are very close with less than ~ 0.02 difference.

Thank you again for your comments. Feel free to submit a pull request and I will be glad to merge it :)