Can't find single merged model

[amala-wilson]

Hello,

I have been trying to execute the script “_exp/expsNumbers.sh” to reproduce the results for the MNIST-SVHN based Numbers dataset. However, I have been running into a few issues and would appreciate it if you could address them.

Based on the results I received, I noticed that there is a merged model in every task. For example, modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar be seen in all task folders. However, based on the diagram you drew in your paper, there should be single merged model after training is done on the server. Currently, I don’t see a single merged model. Rather I see a merged model for every task as seen below.

Is there a single merged model? If so, where is the single merged model located? How can I find it?

MERGED MODELS: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar

This is how my exps_Numbers.sh file looks like:

!/bin/bash

grid="demo"

"# Numbers SETUP" model="MLP_cl_100_100" ds="numbers_nb" # Numbers dataset -> use MLP

############################################ "# TRAIN SERVER" train_script="../train/main_train.py" epochs="10" bs="20"

"# FIM/MAS-IMM/LACL + Task Experts (Common training of server models)" args="--method IMM --lr 0.001 --lmbL2trans 0.001 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS"

args="--method MAS --lr 0.001 --lmb 1 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# EWC"

args="--method EWC --lr 0.001 --lmb 400 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# LWF"

args="--method LWF --lr 0.001 --lmb 1 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

"# JOINT"

args="--method JOINT --lr 0.001 --bs $bs --epochs $epochs --model_name $model --gridsearch_name $grid --ds_name $ds"

./run_wrapper.sh "$train_script $args"

############################################ "# ADAPT/TEST USERS" test_script="../test/main_test.py"

"# FIM-IMM" args="--method IMM --IMM_mode mode --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS-IMM"

args="--method IMM --IMM_mode mode_MAS --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# MAS-LACL"

args="--method LA --LA_mode plain --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# FIM-LACL"

args="--method LA --LA_mode FIM --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# FT (No AdaBN applicable)"

args="--method FT --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds"

"# Same for other methods" "# method="JOINT"" "#method="EWC"" "#method="LWF"" "# method="MAS"" "# args="--method $method --bs 20 --model_name $model --gridsearch_name $grid --ds_name $ds""

./run_wrapper.sh "$test_script $args"

After running the command, “bash exps_Numbers.sh”, I receive the following results:

targets:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]] Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/train_32x32.mat Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/test_32x32.mat TASK numbers=0,1 task_targets=[0, 1]

• TASK numbers=0,1: training
• TASK numbers=0,1: SVHN user
• TASK numbers=0,1: MNIST user TASK numbers=2,3 task_targets=[2, 3]
• TASK numbers=2,3: training
• TASK numbers=2,3: SVHN user
• TASK numbers=2,3: MNIST user TASK numbers=4,5 task_targets=[4, 5]
• TASK numbers=4,5: training
• TASK numbers=4,5: SVHN user
• TASK numbers=4,5: MNIST user TASK numbers=6,7 task_targets=[6, 7]
• TASK numbers=6,7: training
• TASK numbers=6,7: SVHN user
• TASK numbers=6,7: MNIST user TASK numbers=8,9 task_targets=[8, 9]
• TASK numbers=8,9: training
• TASK numbers=8,9: SVHN user
• TASK numbers=8,9: MNIST user Custom MNIST+SVHN preprocessing finished targets:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]] Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/train_32x32.mat Using downloaded and verified file: /scratch1/amala/DUA/data/datasets/SVHN/test_32x32.mat TASK numbers=0,1 task_targets=[0, 1]
• TASK numbers=0,1: training
• TASK numbers=0,1: SVHN user
• TASK numbers=0,1: MNIST user TASK numbers=2,3 task_targets=[2, 3]
• TASK numbers=2,3: training
• TASK numbers=2,3: SVHN user
• TASK numbers=2,3: MNIST user TASK numbers=4,5 task_targets=[4, 5]
• TASK numbers=4,5: training
• TASK numbers=4,5: SVHN user
• TASK numbers=4,5: MNIST user TASK numbers=6,7 task_targets=[6, 7]
• TASK numbers=6,7: training
• TASK numbers=6,7: SVHN user
• TASK numbers=6,7: MNIST user TASK numbers=8,9 task_targets=[8, 9]
• TASK numbers=8,9: training
• TASK numbers=8,9: SVHN user
• TASK numbers=8,9: MNIST user Custom MNIST+SVHN preprocessing finished RUNNING IMM(IMM) in 'train' mode STARTING FROM EXISTING MLP MODEL (name= MLP_cl_100_100 ) in /scratch1/amala/DUA/data/models/MLP_cl_100_100(in_dim=2352).pth.tar RUNNING IMM(IMM) in 'mode' mode <shared_utils.methods.IMM object at 0x14a25fe3d1c0> STARTING TESTING FOR METHOD: IMM_mode => For dir: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo Testing for exp: IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1

USER 1

COLLECTED USER DATASETS: /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth

COLLECTED USER MODELS: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar **** ITERATION 1 ****

IMM preprocessing: 'mode' mode

DATASETS FOR MERGING (IWS, task_agnostic=False): /scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=0,1.pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=2,3.pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=4,5.pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=6,7.pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/tr_trainsubset_SubsetMNIST+SVHN_targetsnumbers=8,9.pth

MODELS FOR MERGING: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar removed reg_params from loaded model removed reg_params from loaded model removed reg_params from loaded model removed reg_params from loaded model MODELS TO MERGE: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/best_model.pth.tar LOADED 5 MODELS in MEMORY IMM PREPROCESSING: Mode mode, overwrite=False *** ESTIMATING IWS on MODEL of TASK 1

• IW: MODEL of TASK 1, on DS of TASK=1 Calculating precision matrix *** ESTIMATING IWS on MODEL of TASK 2
• IW: MODEL of TASK 2, on DS of TASK=2 Calculating precision matrix *** ESTIMATING IWS on MODEL of TASK 3
• IW: MODEL of TASK 3, on DS of TASK=3 Calculating precision matrix *** ESTIMATING IWS on MODEL of TASK 4
• IW: MODEL of TASK 4, on DS of TASK=4 Calculating precision matrix *** ESTIMATING IWS on MODEL of TASK 5

• IW: MODEL of TASK 5, on DS of TASK=5 Calculating precision matrix MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 2 NOT MERGING PARAM classifier.4.weight, as it is a head param name NOT MERGING PARAM classifier.4.bias, as it is a head param name => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 3 NOT MERGING PARAM classifier.4.weight, as it is a head param name NOT MERGING PARAM classifier.4.bias, as it is a head param name => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 4 NOT MERGING PARAM classifier.4.weight, as it is a head param name NOT MERGING PARAM classifier.4.bias, as it is a head param name => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar MERGE MODE=[MEAN=False, IW=True] Merging models for TASK 5 NOT MERGING PARAM classifier.4.weight, as it is a head param name NOT MERGING PARAM classifier.4.bias, as it is a head param name => SAVED MERGED MODEL: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar MERGED MODELS: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar

AFTER PREPROCESSING USER DATASETS: /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth

AFTER PREPROCESSING USER MODELS: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Testing on 5 task datasets TESTING ON TASK 1 Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[0, 1].pth => Testing model trained up to and including TASK 1 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar Overall Accuracy: 94.9137679041216 => Testing model trained up to and including TASK 2 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar Overall Accuracy: 89.09675533469746 => Testing model trained up to and including TASK 3 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar Overall Accuracy: 87.0798012277112 => Testing model trained up to and including TASK 4 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar Overall Accuracy: 85.8228588132125 => Testing model trained up to and including TASK 5 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_1/TASK_TRAINING/best_model.pth.tar Overall Accuracy: 85.09207833966677 TESTING ON TASK 2 Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[2, 3].pth => Testing model trained up to and including TASK 2 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 89.55903271692745 => Testing model trained up to and including TASK 3 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 90.49786628733997 => Testing model trained up to and including TASK 4 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 90.7823613086771 => Testing model trained up to and including TASK 5 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_2/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 91.2375533428165 TESTING ON TASK 3 Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[4, 5].pth => Testing model trained up to and including TASK 3 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 65.14472075010191 => Testing model trained up to and including TASK 4 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 65.87851610273135 => Testing model trained up to and including TASK 5 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_3/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 65.79698328577253 TESTING ON TASK 4 Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[6, 7].pth => Testing model trained up to and including TASK 4 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 56.95695695695696 => Testing model trained up to and including TASK 5 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_4/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 58.85885885885886 TESTING ON TASK 5 Testing Dataset = /scratch1/amala/DUA/data/datasets/MNIST+SVHN/user_testsubset_SubsetSVHN_targets[8, 9].pth => Testing model trained up to and including TASK 5 Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar EVAL on prev heads: /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar Overall Accuracy: 67.91641057160417 FINISHED testing for user: 1 USER1 results: {'acc': {0: [94.9137679041216, 89.09675533469746, 87.0798012277112, 85.8228588132125, 85.09207833966677], 1: [89.55903271692745, 90.49786628733997, 90.7823613086771, 91.2375533428165], 2: [65.14472075010191, 65.87851610273135, 65.79698328577253], 3: [56.95695695695696, 58.85885885885886], 4: [67.91641057160417]}, 'forgetting': {0: [5.817012569424136, 7.833966676410398, 9.090909090909093, 9.821689564454829], 1: [-0.9388335704125268, -1.2233285917496488, -1.6785206258890497], 2: [-0.7337953526294427, -0.6522625356706158], 3: [-1.9019019019019012], 4: []}} **** ITERATION 2 **** [WARN] SKIPPING iteration, as no randomness introduced. **** ITERATION 3 **** [WARN] SKIPPING iteration, as no randomness introduced. **** ITERATION 4 **** [WARN] SKIPPING iteration, as no randomness introduced. **** ITERATION 5 **** [WARN] SKIPPING iteration, as no randomness introduced. Saved exp results to: /scratch1/amala/DUA/results/test/results/NumbersDataset_nb_tasks/IMM_mode/MLP_cl_100_100/demo/subset=[0, 1, 2, 3, 4]/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/evalresults_IMM_mode_user1.pth

[Mattdl]

Hi thanks for your comment! The models are merged incrementally, so if you want the final model you should be looking at the model for task 5.

See also in the logs:

=> Testing model trained up to and including TASK 5
    Testing on model = /scratch1/amala/DUA/results/train/NumbersDataset_nb_tasks/IMM/MLP_cl_100_100/gridsearch/demo/IMM_lr=0.001_bs=20_epochs=10_lmbL2trans=0.001_alpha=1/task_5/TASK_TRAINING/modelmerge_user1_mode_DS=NumbersDataset_nb_tasks_id=23f26144-1730-11ec-806c-506b4b3bff98.pth.tar

Meaning that this model is 'trained up to and including TASK 5', hence all tasks. Hope this helps you out!

[amala-wilson]

Thank you for your response. So is it okay to confirm that the "single merged model M^" (referenced from your paper) that will be sent to the user device is the merged model that is located in the "...task_5/TASK_TRAINING/" folder?

Also, my assumption is that this final model should be able to detect any images that are passed to it. For example, if I pass in an image that contains "1", the model should be able to detect it as 1. However, there are only two output nodes in the merged model so how can it detect any number that's being sent? Shouldn't the output nodes be 10 in the merged model?