iCaRL's accuracy different from 'Three scenarios for continual learning'

[JosephKJ]

I have ran the code in the repo with just one change: in _compare.py: changed 'scenario' to 'class'.

The results is as follows

############################################################
SUMMARY RESULTS: splitMNIST  -  class-incremental learning
------------------------------------------------------------
None         19.94
EWC          19.93
o-EWC        19.91
SI           19.95
LwF          24.00
DGR          91.69
DGR+distil   90.42
iCaRL        72.83
Offline      97.59
############################################################

I tried with two seeds (1, 99) both gave ~72 for iCaRL. The reported accuracy in the paper is ~94.

Am I missing something that is leading to this discrepancy?

[GMvandeVen]

Thank you for pointing this out! You're right, there was a mistake in the code. The stored exemplars were not actually being added to the training data. They were added to [training_dataset], but then erroneously [train_dataset] was used to train on. The change I made to fix this was to replace [train_dataset] by [training_dataset] in line 117 of train.py.

The version of the code used to obtain the results reported in the paper did not have this mistake, hence the discrepancy. Sorry for the mistake and thank you for pointing it out!

[JosephKJ]

Thank you for your quick response.

I have verified your fix. Thanks!

############################################################
SUMMARY RESULTS: splitMNIST  -  class-incremental learning
------------------------------------------------------------
None         19.91
EWC          20.49
o-EWC        20.66
SI           19.91
LwF          23.60
DGR          91.63
DGR+distil   93.46
iCaRL        95.08
Offline      98.15
############################################################