search

AlbinSou / ocl_survey

Code for "A Comprehensive Empirical Evaluation on Online Continual Learning" ICCVW 2023 VCL Workshop
https://arxiv.org/abs/2308.10328
MIT License
31 stars 9 forks source link

can reproduce er result, but cannot reproduce scr result #11

Closed zhaoedf closed 5 months ago

zhaoedf commented 5 months ago

For efficiency, i only run strategy ER, GDumb(OOM), SCR. The ER result can be reproduced as the paper. However, the scr result is unexpected low as listed below. Are there any hyper-params incorrectly beint set for SCR? i provide my yaml config below. Thanks~

# ER result
-- >> End of eval phase << --
    CumulativeAccuracy/eval_phase/test_stream/Exp000 = 0.6660
    CumulativeAccuracy/eval_phase/test_stream/Exp001 = 0.6590
    CumulativeAccuracy/eval_phase/test_stream/Exp002 = 0.5780
    CumulativeAccuracy/eval_phase/test_stream/Exp003 = 0.5680
    CumulativeAccuracy/eval_phase/test_stream/Exp004 = 0.5356
    CumulativeAccuracy/eval_phase/test_stream/Exp005 = 0.5037
    CumulativeAccuracy/eval_phase/test_stream/Exp006 = 0.4677
    CumulativeAccuracy/eval_phase/test_stream/Exp007 = 0.4510
    CumulativeAccuracy/eval_phase/test_stream/Exp008 = 0.4353
    CumulativeAccuracy/eval_phase/test_stream/Exp009 = 0.4314
    CumulativeAccuracy/eval_phase/test_stream/Exp010 = 0.4158
    CumulativeAccuracy/eval_phase/test_stream/Exp011 = 0.4058
    CumulativeAccuracy/eval_phase/test_stream/Exp012 = 0.3863
    CumulativeAccuracy/eval_phase/test_stream/Exp013 = 0.3749
    CumulativeAccuracy/eval_phase/test_stream/Exp014 = 0.3673
    CumulativeAccuracy/eval_phase/test_stream/Exp015 = 0.3554
    CumulativeAccuracy/eval_phase/test_stream/Exp016 = 0.3505
    CumulativeAccuracy/eval_phase/test_stream/Exp017 = 0.3481
    CumulativeAccuracy/eval_phase/test_stream/Exp018 = 0.3338
    CumulativeAccuracy/eval_phase/test_stream/Exp019 = 0.2820
    Loss_Stream/eval_phase/test_stream/Task000 = 4.3728
    Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.2820
# SCR result
-- >> End of eval phase << --
    CumulativeAccuracy/eval_phase/test_stream/Exp000 = 0.5300
    CumulativeAccuracy/eval_phase/test_stream/Exp001 = 0.4610
    CumulativeAccuracy/eval_phase/test_stream/Exp002 = 0.3460
    CumulativeAccuracy/eval_phase/test_stream/Exp003 = 0.2770
    CumulativeAccuracy/eval_phase/test_stream/Exp004 = 0.2276
    CumulativeAccuracy/eval_phase/test_stream/Exp005 = 0.1993
    CumulativeAccuracy/eval_phase/test_stream/Exp006 = 0.1731
    CumulativeAccuracy/eval_phase/test_stream/Exp007 = 0.1568
    CumulativeAccuracy/eval_phase/test_stream/Exp008 = 0.1427
    CumulativeAccuracy/eval_phase/test_stream/Exp009 = 0.1338
    CumulativeAccuracy/eval_phase/test_stream/Exp010 = 0.1235
    CumulativeAccuracy/eval_phase/test_stream/Exp011 = 0.1202
    CumulativeAccuracy/eval_phase/test_stream/Exp012 = 0.1112
    CumulativeAccuracy/eval_phase/test_stream/Exp013 = 0.1040
    CumulativeAccuracy/eval_phase/test_stream/Exp014 = 0.0972
    CumulativeAccuracy/eval_phase/test_stream/Exp015 = 0.0909
    CumulativeAccuracy/eval_phase/test_stream/Exp016 = 0.0861
    CumulativeAccuracy/eval_phase/test_stream/Exp017 = 0.0812
    CumulativeAccuracy/eval_phase/test_stream/Exp018 = 0.0801
    CumulativeAccuracy/eval_phase/test_stream/Exp019 = 0.0761
    Loss_Stream/eval_phase/test_stream/Task000 = 4.4721
    Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.0761
# yaml config for SCR
benchmark:
  factory_args:
    return_task_id: false
    shuffle: true
    class_ids_from_zero_in_each_exp: false
    class_ids_from_zero_from_first_exp: true
    benchmark_name: split_cifar100
    n_experiences: 20
    val_size: 0.0
    use_transforms: false
    fixed_class_order: null
  dataset_name: cifar100
  dataset_root: /data2/defeng_data/ocil/ocl_survey_data
scheduler:
  type: null
optimizer:
  type: SGD
  lr: 0.1
  momentum: 0.0
  weight_decay: 0.0
model:
  model_type: slim_resnet18
strategy:
  mem_size: 2000
  batch_size_mem: 118
  train_mb_size: 10
  eval_mb_size: 128
  train_epochs: 3
  device: cuda
  use_task_boundaries: false
  name: scr
  temperature: 1.0
evaluation:
  peval_mode: experience
  eval_every: -1
  metrics:
  - accuracy_metrics
  - cumulative_accuracy
  - loss_metrics
  - clock
  - time
  loggers_strategy:
  - interactive
  - text
  - tensorboard
  - json
  num_gpus: 1
  num_cpus: 8
  num_actors: 3
experiment:
  results_root: ./results
  seed: 0
  save_models: false
  train_online: true
  logdir: ./results/scr_split_cifar100_20_2000/0
  debug: false

# running command
python main.py experiment=split_cifar100 strategy=scr
AlbinSou commented 5 months ago

Hello, can you try to run python main.py experiment=split_cifar100 strategy=scr +best_configs=split_cifar100/scr.yaml ? This will use the best config found via hyperparameter search, which is the one we used in our experiments.

zhaoedf commented 5 months ago

Hello, can you try to run python main.py experiment=split_cifar100 strategy=scr +best_configs=split_cifar100/scr.yaml ? This will use the best config found via hyperparameter search, which is the one we used in our experiments.

ok, i will try.