About accuracy

[daihu-ye]

In your paper , you said you run the experiment three times,did you use different random seed,or you just use seed 1993 and run 3 times?I found that different random seed may affect the final accuracy.So i want to know your configuration for fair comparison.

[yaoyao-liu]

Thanks for your interest in our work.

We use the same seed (1993) to generate the class order and use different seeds to run the following parts of the experiments.

[daihu-ye]

I don't understand.The seed is used to generate the class order in your code,3 different seeds mean 3 class order,right? I just run python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100 python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1994 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100 python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1996 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100 Then I calulate averages ± standard deviations.The results is the final average accuracy when N=5. Is it right? I also did some experiments and found that different seeds may affect the final accuracy(sometimes more than 1%). And in icarl experiments,with AANets,i can't get the same results as your paper (your paper report 64.22 when N=5),but i only got 62.

[yaoyao-liu]

You need to edit this function to run experiments with the same class order and different random seeds: https://github.com/yaoyao-liu/class-incremental-learning/blob/834de4639833641ee921e50b2d180b1b67bb0128/adaptive-aggregation-networks/trainer/base_trainer.py#L214-L241 I'll check the results of iCaRL+AANets. Could you please send me the command you use to run iCaRL+AANets?

[daihu-ye]

Thanks for your explanation.Just edit the path of the class order file,let it lead to the same class order file (seed 1993 generate ). It seems that the random seed is only used for selecting the exemplars in https://github.com/yaoyao-liu/class-incremental-learning/blob/main/adaptive-aggregation-networks/trainer/base_trainer.py in function gen_balanced_loader the_idx = np.random.randint(0,len(X_train_this_step),size=self.args.nb_cl*self.args.nb_protos) if we already get the class order file ,right?

[yaoyao-liu]

I think the random seed will influence many steps in the following code.

[daihu-ye]

Maybe i should run the experiments again.The result is actually got by 3 class orders ,it may influence the final accuracy. Thanks for your response.