CNN or NME?

[nttung1110]

Hi,

Thanks for your interesting work! I would love to know if the reproduced score demonstrated in your paper is CNN top 1 curve or NME top1 curve in the provided code? I tried to run your code without changing anything and got the following results for three CIL method: bic, icarl, and foster. The experiments are run on CIFAR100 dataset with 10 incremental classes for each task

bic: 2023-05-23 01:51:11,906 [trainer.py] => CNN: {'total': 39.06, '00-09': 53.9, '10-19': 36.5, '20-29': 47.1, '30-39': 39.9, '40-49': 51.6, '50-59': 42.1, '60-69': 0.0, '70-79': 11.5, '80-89': 46.4, '90-99': 61.6, 'old': 36.56, 'new': 61.6} 2023-05-23 01:51:11,906 [trainer.py] => NME: {'total': 46.11, '00-09': 49.6, '10-19': 32.3, '20-29': 50.6, '30-39': 38.5, '40-49': 50.4, '50-59': 38.4, '60-69': 46.0, '70-79': 45.5, '80-89': 52.6, '90-99': 57.2, 'old': 44.88, 'new': 57.2} 2023-05-23 01:51:11,906 [trainer.py] => CNN top1 curve: [86.5, 71.3, 68.4, 62.35, 61.42, 58.17, 48.27, 42.31, 40.74, 39.06] 2023-05-23 01:51:11,906 [trainer.py] => CNN top5 curve: [99.0, 94.25, 91.77, 88.98, 87.98, 86.15, 72.44, 70.03, 68.52, 66.57] 2023-05-23 01:51:11,906 [trainer.py] => NME top1 curve: [86.3, 71.6, 68.73, 62.7, 61.72, 57.62, 55.69, 50.46, 48.24, 46.11] 2023-05-23 01:51:11,906 [trainer.py] => NME top5 curve: [98.7, 94.0, 91.43, 88.88, 87.56, 84.62, 82.54, 79.0, 77.08, 75.35]

icarl: 2023-05-23 00:26:06,251 [trainer.py] => CNN: {'total': 42.06, '00-09': 35.4, '10-19': 23.3, '20-29': 34.9, '30-39': 29.6, '40-49': 38.7, '50-59': 29.2, '60-69': 43.1, '70-79': 44.5, '80-89': 60.9, '90-99': 81.0, 'old': 37.73, 'new': 81.0} 2023-05-23 00:26:06,251 [trainer.py] => NME: {'total': 49.4, '00-09': 47.4, '10-19': 32.0, '20-29': 48.5, '30-39': 40.2, '40-49': 51.3, '50-59': 42.1, '60-69': 55.2, '70-79': 53.2, '80-89': 59.3, '90-99': 64.8, 'old': 47.69, 'new': 64.8} 2023-05-23 00:26:06,251 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17, 52.29, 47.32, 44.53, 42.06] 2023-05-23 00:26:06,251 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02, 80.8, 77.85, 75.43, 72.11] 2023-05-23 00:26:06,251 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77, 58.09, 54.55, 51.34, 49.4] 2023-05-23 00:26:06,251 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45, 83.87, 82.19, 79.02, 77.53]

foster: 2023-05-23 01:18:52,589 [trainer.py] => CNN: {'total': 54.55, '00-09': 48.8, '10-19': 33.5, '20-29': 51.3, '30-39': 46.8, '40-49': 59.0, '50-59': 52.7, '60-69': 66.9, '70-79': 60.5, '80-89': 64.0, '90-99': 62.0, 'old': 53.72, 'new': 62.0} 2023-05-23 01:18:52,589 [trainer.py] => NME: {'total': 48.13, '00-09': 46.9, '10-19': 32.9, '20-29': 49.5, '30-39': 42.9, '40-49': 52.0, '50-59': 42.4, '60-69': 50.2, '70-79': 43.1, '80-89': 45.6, '90-99': 75.8, 'old': 45.06, 'new': 75.8} 2023-05-23 01:18:52,589 [trainer.py] => CNN top1 curve: [90.8, 81.2, 76.27, 67.72, 63.86, 60.65, 59.81, 56.8, 55.58, 54.55] 2023-05-23 01:18:52,589 [trainer.py] => CNN top5 curve: [99.2, 95.75, 93.83, 90.25, 87.82, 85.35, 84.29, 83.19, 82.04, 81.68] 2023-05-23 01:18:52,589 [trainer.py] => NME top1 curve: [90.2, 82.35, 77.07, 68.8, 64.28, 59.82, 57.11, 51.94, 51.09, 48.13] 2023-05-23 01:18:52,589 [trainer.py] => NME top5 curve: [99.2, 96.85, 95.03, 91.02, 88.38, 85.8, 83.57, 81.66, 79.91, 76.7]

In addition, I found that score of bic at task '60-69' is almost 0 which indicate that the learning algorithm lost its capability in predicting for that task. I don't know if this is normal for bic method or I have encountered some issues when training from my side. Thanks for your help!

[zhoudw-zdw]

Hi,

For those mentioned "we use NME classifier" in the paper (e.g., [1][2]), we report the NME accuracy.

For others, we report CNN accuracy.

I am not quite sure about your bug, maybe you should attach your json setting for bic.

[1] iCaRL: Incremental Classifier and Representation Learning [2] Co-Transport for Class-Incremental Learning

[nttung1110]

Hi @zhoudw-zdw ,

Thanks for your reply. Here is json setting that I've used for bic experiment

{ "prefix": "reproduce", "dataset": "cifar100", "memory_size": 2000, "memory_per_class": 20, "fixed_memory": false, "shuffle": true, "init_cls": 10, "increment": 10, "model_name": "bic", "convnet_type": "resnet32", "device": ["2"], "seed": [1993] }

[zhoudw-zdw]

It seems correct, and the training results are somehow normal (except its results are lower than icarl).

I did not find the "score of bic at task '60-69' is almost 0", where did it happen?

BTW, you should check the algorithm-specific params in bic and make it cosistent with this repo.

[nttung1110]

Hi @zhoudw-zdw,

Thanks for your instant reply. Yes, I'm pretty sure that the algorithm-specific params in bic is the same as the original code of this repo. About the score of bic at task '60-69', you can see in the results that I showed above

bic: 2023-05-23 01:51:11,906 [trainer.py] => CNN: {'total': 39.06, '00-09': 53.9, '10-19': 36.5, '20-29': 47.1, '30-39': 39.9, '40-49': 51.6, '50-59': 42.1, '60-69': 0.0, '70-79': 11.5, '80-89': 46.4, '90-99': 61.6, 'old': 36.56, 'new': 61.6}