jhao6
commented
3 months ago I am sorry for misleading you for the default hyperparameter settings. There are several important hyperparameter settings in our experiments, including "--outer_iter" for outer optimization loops for sample weights, and "--inner_iter" for inner optimization loops for model parameters. We actually set "--outer_iter=10" for SplitCIFAR100 in our experiments rather than 5 default set in "main.py". You can find the details of hyperparameter setting in Table 7, page 20 (Appendix) of our paper. We use bilevel optimization algorithm to update the sample weights and model parameters, so the number of outer loops ("--outer_iter") and inner loops ( "--inner_iter") determines the selection of coresets and the convergence results. The corresponding learning rates for inner and outer loops also have key impacts on the results. Our settings for theses hyperparameters may not be the best combinations, you could tune them carefully for your data and tasks. Another hyperparameter you should pay attention to is the "Q loops" (line 77 in bcsr_training.py), which is responsible for approximating the hessian-inverse-vector product. We set it to 3 by default, but feel free to tune it if necessary, typically less than 10.
I am trying to reproduce results for SplitCIFAR-100, I used the exact same hyperparameters mentioned in the code. I am getting average accuracy as 59.42, and forgetting as 6.41. In the paper,average accuracy is mentioned as 61.60. Could you please provide hyperparameter settings for reproducing results?