mmasana
commented
3 years ago Hi, We provided lots of comments and links to help people understand better how the approaches work, so very happy to hear you found it useful! Our reported results come from our implementation of LUCIR after running it for the iCaRL seed 10 times, obtaining the following accuracies:
[43.9, 50.0, 43.4, 43.9, 41.3, 46.3, 42.2, 38.8, 44.1, 39.8] --> mean: 43.4, std: 3.0the main differences with LUCIR's results come from the following:
- LR policy and epochs. LUCIR starts with a LR of 0.1 that is divided by 10 at epochs 80 and 120. The total number of epochs is fixed at 160. Those are values that work well for that scenario with that number of tasks. Our implementation uses the Continual Hyperparameter Search (CHS) which uses a patience scheme to decide which learning rate to use and has a maximum number of 200 epochs (see gridsearch_config.py). We consider our use of the LR more realistic and comparable with other methods, since it takes the decisions at each task without knowledge of other tasks.
- Lambda_base. similarly, they fix lambda_base to be 5 for all the LUCIR experiments on CIFAR-100. Since lambda_base is the main stability-plasticity trade-off, we search for the best trade-off after each task without knowledge of other tasks via the CHS. This has a large effect in the accuracy and forgetting metrics.
- Class ordering. We fix the order in which classes are learned to the one proposed by iCaRL for that experiment in particular. The original LUCIR implementation uses a random class ordering. Since in this scenario half of the classes are learned as the first task, this has quite an impact on the results. For more info on class ordering, you can check our paper "On Class Orderings for Incremental Learning".
In conclusion, the results of our framework should be within the specs of the LUCIR ones (we tested this more than a year ago, so cannot provide results right now) if the LR schedule is set to the one from LUCIR, with random class ordering and the proposed hyperparameters fixed for all the experiment (instead of learned per task without knowledge of future tasks).
Kingofolk
Hello! Thank you for your great work, from which I've learnd a lot. Recently I want to reproduce LUCIR but find that the corresbonding results are quite different from those reported by Hou in his paper (Hou reported an avg acc of 60.18 and yours is 43.4 under the config of 50/11). Which should I follow? The related results are showed as following. your work----------------------------------
Hou's work (LUCIR)-------------------------
And there are also some work reported the avg acc of lucir under the same config. PODNet. ECCV2020-------------------------------
Looking forward for your reply! best wishes!