search

yolky / RFAD

Code for the paper "Efficient Dataset Distillation using Random Feature Approximation"
36 stars 4 forks source link

Reproduce performance w.r.t. CIFAR-10 (IPC=1) #2

Closed LeavesLei closed 1 year ago

LeavesLei commented 1 year ago

Hi, Noel, thanks for the excellent job on dataset distillation.

I attemp to use the repo as the pipeline to distill some datasets. I've tried to distil synthetic images from CIFAR-10 wth ipc=1 by running run_distillation.py, and then the distilled images are evaluatd on finite ConvNet by python eval_distilled_set.py --dataset cifar10 --lr 1e-3 --weight_deca 1e-3 --label_scale 8 --centering --run_finite. However, the final test accuracy is only $34.39%$, which is far away from $53.6$ in the original paper. Do I missed other details?

BTW, there is a small typo in run_dstillation.py:https://github.com/yolky/RFAD/blob/3e1e2e779f1f8726fa8d0b534e2ae1b9bfcab7d1/eval_distilled_set.py#L129 hung_factor should be replaced with label_scale_factor?

Best, Shiye

yolky commented 1 year ago

Hi Shiye,

Thanks for noticing the typo and the reproducing problem. We've updated the repo, and it should resolve the problem. We uploaded the wrong cifar-10 1 ipc distilled dataset in the previous version.

Were you using the uploaded cifar10 example distilled datasets in distilled_images_final or ones that you made yourself?

If you were using the uploaded ones, try running the command (on the updated repo):

python3 eval_distilled_set.py --dataset cifar10 --save_path ./distilled_images_final/cifar10/1/ --run_finite --centering --use_best_hypers

Note that we uploaded the wrong distilled dataset for cifar-10 1 ipc, and have since corrected it. The command "use_best_hypers" should load the set of hyperparameters we used in the paper, which were label_scale = 8, lr = 1e-3, weight_decay = 0, for this particular dataset and ipc. Some of these best hyparameters are quite brittle - particularly, label scaling and the learning rate. You could find the set of hyperparameters used in the file utils.py

We found that performance can change dramatically when changing the label scaling parameter, and it is indeed one of the weaknesses of our approach, since good transfer from the infinite-width regime to the finite-width regime is not always guaranteed.

Also note that the example distilled datasets are with no label learning, so performance may slightly differ from the paper which uses label learning for some of the finite network results. We hopefully get around to uploaded all of our distilled datasets in the next week.

If you distilled your own distilled dataset, may you provide the command you used to generate that distilled dataset?

Noel

LeavesLei commented 1 year ago

Hi, Noel, thanks for this detail reply, and it clearifies my confusions. I will try the latest code later. The commond I used to distill data is as follows:

python run_distillation.py --platt --dataset cifar10

Best, Shiye

yolky commented 1 year ago

Hi,

Did you have a chance to try the latest code? Also, did you have any issues with the infinite width KRR performance? Or was it just with finite width?

Noel

LeavesLei commented 1 year ago

Hi, Noel, sorry for the late reply. I haven't tried the latest code yet due to to many courses in this semester... but I looked the update and think it would work for me. I think the issue can be closed and thanks for your updated codes again:)

Best, Shiye