Request for a template for evaluating other sotas

[genema]

Thanks for the share of the CL study frame! However I wonder how to use iCarl with the framework. I have modified the main_tinyimagenet.sh file via adding static_hyperparams term to the arguments, but still got several errors. Could you please provide a shell file for running iCarl ? Appreciate that.

[genema]

Here is the script I modified. Thnx.
#!/usr/bin/env bash

# DEMO showing how to get survey results
# First make sure your project is configured: See /README.md
# Install requirements, config project paths, data preparation of TinyImagenet
MY_PYTHON=python
EXEC=./framework/main.py

# Add root path
this_script_path=$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P) # Change location to current script
root_path=${this_script_path}/../src/
export PYTHONPATH=$PYTHONPATH:$root_path

echo ">>main.sh>> Project src root '${root_path}' added to Python path"

# params
gridsearch_name='reproduce'      # NAME IT ! Give arbitrary gridsearch name
ds_name='tiny'                  # Dataset, e.g. {tiny, tinyhardeasy, inat, inatrelunrel, recogseq}
model='base_VGG9_cl_512_512'    # Other survey models:
                                # iNaturalist:{alexnet_pretrained},
                                # TinyImagenet: {base: base_VGG9_cl_512_512,
                                #                wide: wide_VGG9_cl_512_512,
                                #                deep: deep_VGG22_cl_512_512}

lr_grid='1e-2,5e-3,1e-3,5e-4,1e-4'
boot_lr_grid='1e-1,5e-2,1e-2,5e-3,1e-3,5e-4,1e-4'
# Lookup default (static) hyperparameters in methods/method.py

# First task modeldump for SI
method_name='SI'  # 
$MY_PYTHON $EXEC --runmode "first_task_basemodel_dump" \
                 --gridsearch_name ${gridsearch_name} \
                 --method_name ${method_name} \
                 --ds_name ${ds_name} \
                 --lr_grid ${lr_grid} \
                 --boot_lr_grid ${boot_lr_grid} \
                 ${model}

echo ">>main.sh>> first task model dump for SI done."
# Run any method using this first task model (enable testing by --test argument)
method_name='ICARL' # Choose any method name (see methods/method.py for method names)
echo ">>main.sh>> start using '${method_name}' for training."
$MY_PYTHON $EXEC --gridsearch_name ${gridsearch_name}\
                 --method_name ${method_name} \
                 --ds_name ${ds_name} \
                 --lr_grid ${lr_grid} --boot_lr_grid ${boot_lr_grid} \
                 --static_hyperparams 'def' --test ${model}

# Methods with possible deviations from default hyperparam values
#method_name='EBLL'
#$MY_PYTHON $EXEC --gridsearch_name ${gridsearch_name} --method_name ${method_name} --ds_name ${ds_name} \
#                 --static_hyperparams 'def;def;1e-1,1e-2;100,300' --lr_grid ${lr_grid} --boot_lr_grid ${boot_lr_grid} \
#                 --test ${model}

# Hardcoded Plot Config (Specific for original demo setup)
# Change plot config file to your exps
python_filename="./utilities/plot_configs/demo.py"
$MY_PYTHON  "$python_filename"

[genema]

I notice that for iCarl whose "args.wrap_first_task_model" is True, the value of 'skip_to_the_post' is always True in framework_train.py, which will skip the parts including manager.best_model_path initialization. I am not sure if my usage above is correct.

[genema]

Since the iCarl implementation in CLsurvey frame is modified using the implementation of GEM, both GEM and iCarl methods result in errors.

It seems that the problematic part is in the poststep() of both GEM and iCarl in method.py. As the flag 'skip_to_post' is always True (specified as a class attribution), it will get into the poststep() function immediatly. However, the lr/ heurisric_exp_dir / best_model_path attributions of manager instance are initialized in the PHASE 1 & 2 in the hyperparam-framework.