search

vikasverma1077 / manifold_mixup

Code for reproducing Manifold Mixup results (ICML 2019)
486 stars 62 forks source link

Reproducing CIFAR10 supervised results #5

Open wtqin382 opened 5 years ago

wtqin382 commented 5 years ago

I am attempting to reproduce your CIFAR10 supervised results from Table 1 (https://arxiv.org/pdf/1806.05236.pdf) using code from this repository. I cannot get within 0.5% of the following results:

For example, the paper is vague on details such as initial learning rate, batchsize, Nesterov or not, and other settings. Could you kindly provide command line invocations to reproduce those results?

Also when running your training code I see test error variation of around 0.3-0.5% from epoch to epoch. Do you report results over multiple seeds or are these figures single-seed estimates of the test error?

alexmlamb commented 5 years ago

Hello,

Thanks for reaching out to us. How many epochs were you running for? I think that this has been the most common issue.

We can also give an exact command line command, at least for PreActResNet18.

We also submitted a new draft to arxiv which makes the experimental setup clearer and now reports mean/standard deviation across 5 trials for these experiments.

Best,

Alex.

alexmlamb commented 5 years ago

Hello,

Can you give us more details on what you ran?

Best,

Alex.

vikasverma1077 commented 5 years ago

@wtqin382 if you can paste the output log, I can see what is going wrong.

"For example, the paper is vague on details such as initial learning rate, batchsize, Nesterov or not, and other settings." All these details are present in the Appendix of the paper (since the first version), except the Nesterov ( which is "True" by default in the code)

vikasverma1077 commented 5 years ago

Hi @wtqin382, we have uploaded the new code with the command line argument to run each of the experiments in the paper. It will be great if you can try to reproduce the results and let us know if there is some issues.

vikasverma1077 commented 5 years ago

@wtqin382 Hi there! were you able to reproduce the results?