Configuration to reproduce results in appendix F.1

[smonsays]

I am wondering what configuration was used to produce the results for Split MNIST (incremental domain learning) reported in Table 5. The paper mentions:

For task learning on Split MNIST we used STD init of 0.02, for domain learning we used 0.05 [...]

However, in run_exps.sh only the following configuration is mentioned for incremental domain learning on Split MNIST, which uses a std_init=0.02:

## Discrete task agnostic on Split MNIST (10 epochs per task) domain learning
CUDA_VISIBLE_DEVICES=3 python main.py --logname discrete_domain_split_mnist_5tasks_4epochs_seed${seed} --nn_arch mnist_simple_net_400width_domainlearning_1024input_2cls_1ds --test_freq 10 --seed ${seed} --permute_seed ${seed} --dataset ds_padded_split_mnist --num_epochs $(( 5 * 4 )) --optimizer bgd --std_init 0.02 --batch_size 128 --results_dir split_domain_1000width_10epc --train_mc_iters 10 --inference_mc --test_mc_iters 10 --inference_map --inference_committee --committee_size 10 --inference_aggsoftmax --inference_initstd --desc desc

Similarly, I am wondering what configuration was used to produce the results for Permuted MNIST (incremental domain learning) reported in Table 6. I am surprised to see that the accuracy reported in Table 6 (84.40% ± 1.51) seems to differ quite a bit from the performance shown in Figure 4 (~90.00%). Are these different configurations?

[igolan]

Hi, run_exps.sh only provides examples, it does not contain all the experiments of the paper. To reproduce an experiment, you might have to adjust some hyper-parameters to match those reported in the paper (for example, changing the std_init for in the example you mentioned. If you are having trouble with some hyper-parameters, contact us.

As for figure 4 and table 6: Figure 4 experiments are with 300 epochs per task. Table 6 experiments are with 10 epochs per task.