search

mmasana / FACIL

Framework for Analysis of Class-Incremental Learning with 12 state-of-the-art methods and 3 baselines.
https://arxiv.org/pdf/2010.15277.pdf
MIT License
512 stars 98 forks source link

LUCIR in memoryless mode #3

Closed HabibSlim closed 3 years ago

HabibSlim commented 3 years ago

In the code for LUCIR, we can read the following comment:

LUCIR is expected to be used with exemplars. If needed to be used without exemplars, overwrite here the _get_optimizer function with the one in LwF and update the criterion

How should the criterion function be updated to allow for a memoryless scenario? It is my understanding that the criterion function should already work without any exemplars in accordance to the original paper, but maybe I'm missing a detail.

HabibSlim commented 3 years ago

(besides the fact that the margin ranking loss is unappliable of course)

mmasana commented 3 years ago

We didn't test out the usage of LUCIR without exemplars, since the original paper makes use of them (I think they use at least 1 sample per class in all experiments, including the one measuring the effect of the memory size). In general, when a method would be used without exemplars, it would obtain better results if the access to previous heads is restricted during training of new tasks (specially if weight decay is being used). However, in LUCIR's original code, the less-forget loss constraint is already applied without modifying other heads already (as seen in the _get_optimizer() function).

Regarding the criterion, I agree that the margin ranking loss has to be removed (by using argument --remove-margin-ranking) and the current criterion should work already as it is provided. However, I would have to run a couple of experiments to make sure it doesn't break somewhere else.

Therefore, I believe that the comment in src/approach/lucir.py should be replaced by smth like:

LUCIR is expected to be used with exemplars. If needed to be used without exemplars, make sure to use the `--remove-margin-ranking` argument.
HabibSlim commented 3 years ago

Thank you very much for the useful clarifications! I'm going to proceed as you suggested and explicitly disable the margin ranking loss when running the code without exemplars.