search

mmasana / FACIL

Framework for Analysis of Class-Incremental Learning with 12 state-of-the-art methods and 3 baselines.
https://arxiv.org/pdf/2010.15277.pdf
MIT License
512 stars 98 forks source link

UNABLE TO MATCH THE ACCURACY RESULTS FOR CIFAR100 DATASET FOR LWF APPROACH #39

Closed arnabphoenix closed 1 year ago

arnabphoenix commented 1 year ago

============================================================================================================ Arguments = approach: lwf batch_size: 64 clipping: 10000 datasets: ['cifar100'] eval_on_train: False exp_name: None fix_bn: False gpu: 0 gridsearch_tasks: -1 keep_existing_head: False last_layer_analysis: False log: ['disk'] lr: 0.1 lr_factor: 3 lr_min: 0.0001 lr_patience: 5 momentum: 0.0 multi_softmax: False nc_first_task: None nepochs: 200 network: resnet32 no_cudnn_deterministic: False num_tasks: 4 num_workers: 4 pin_memory: False pretrained: False results_path: ../results save_models: False seed: 0 stop_at_task: 0 use_valid_only: False warmup_lr_factor: 1.0 warmup_nepochs: 0 weight_decay: 0.0

Approach arguments = T: 2 lamb: 1

Exemplars dataset arguments = exemplar_selection: random num_exemplars: 0 num_exemplars_per_class: 0

[(0, 25), (1, 25), (2, 25), (3, 25)] ACCURACY COMING IS

Test on task 0 : loss=1.428 | TAw acc= 54.5%, forg= 9.8%| TAg acc= 6.6%, forg= 57.8% <<< Test on task 1 : loss=3.908 | TAw acc= 59.6%, forg= 8.0%| TAg acc= 14.5%, forg= 47.4% <<< Test on task 2 : loss=4.482 | TAw acc= 59.2%, forg= 5.6%| TAg acc= 21.5%, forg= 35.2% <<< Test on task 3 : loss=4.317 | TAw acc= 70.9%, forg= 0.0%| TAg acc= 67.8%, forg= 0.0% <<< Save at ../results/cifar100_lwf

TAw Acc 64.3% 0.0% 0.0% 0.0% Avg.: 64.3% 61.3% 67.7% 0.0% 0.0% Avg.: 64.5% 57.5% 63.8% 64.8% 0.0% Avg.: 62.1% 54.5% 59.6% 59.2% 70.9% Avg.: 61.1%

TAg Acc 64.3% 0.0% 0.0% 0.0% Avg.: 64.3% 39.6% 61.9% 0.0% 0.0% Avg.: 50.8% 21.9% 39.7% 56.7% 0.0% Avg.: 39.4% 6.6% 14.5% 21.5% 67.8% Avg.: 27.6%

TAw Forg 0.0% 0.0% 0.0% 0.0% 3.0% 0.0% 0.0% 0.0% Avg.: 3.0% 6.8% 3.8% 0.0% 0.0% Avg.: 5.3% 9.8% 8.0% 5.6% 0.0% Avg.: 7.8%

TAg Forg 0.0% 0.0% 0.0% 0.0% 24.7% 0.0% 0.0% 0.0% Avg.: 24.7% 42.4% 22.2% 0.0% 0.0% Avg.: 32.3% 57.8% 47.4% 35.2% 0.0% Avg.: 46.8%

PLEASE ALSO TELL ME WHAT IS TAWFORG AND TAGFORG
mmasana commented 1 year ago

The experiment you are running is CIFAR-100 with 4 tasks of 25 classes each, and not on the same proposed ordering as in the paper. In our survey we provided results for 10 tasks of 10 classes and for 11 tasks with 50 classes first and then 5 classes for the rest. Therefore, not sure what you want to compare to. Please provide more context.

TAw Forg is task-aware forgetting. TAg Forg is task-agnostic forgetting.

arnabphoenix commented 1 year ago

Respected Sir, Thank You very much for clarifying and giving the direction. Now the accuracies are matching