Finetune on Indoor67 gives very low accuracy

[ziqi-zhang]

Hi, Thanks for sharing the code. I run the retraining and fine-tuning scripts on Indoor67 but got very low accuracy. I used the setting in scripts/Re-training.sh for re-training and the same setting except --reinit for fine-tuning. The fine-tuned accuracy after 30k iterations is 50% and re-trained accuracy after 30k iterations is 44.4. The results are far behind the results in the paper (75% and 59%). Can you help to check the result?

Here is the detailed retrain accuracy of every 1k iterations

iter loss Acc 0 4.21 1.49 1000 2.99 23.36 2000 2.68 28.88 3000 3.06 23.21 4000 2.34 38.73 5000 2.58 34.85 6000 2.35 38.96 7000 2.28 42.16 8000 2.52 35.9 9000 2.25 41.19 10000 2.49 38.36 11000 2.26 41.94 12000 2.25 43.58 13000 2.18 43.36 14000 2.26 41.79 15000 2.58 37.99 16000 2.32 40.6 17000 2.41 39.55 18000 2.36 40.37 19000 2.31 40.82 20000 2.38 41.87 21000 2.32 41.79 22000 2.19 44.25 23000 2.25 44.55 24000 2.17 44.55 25000 2.1 45.6 26000 2.3 43.28 27000 2.41 41.57 28000 2.09 45.15 29000 2.43 42.01 30000 2.19 44.4

Thanks, Ziqi

[ziqi-zhang]

And here is the retrain script:

#!/bin/bash

iter=90000
id=1
splmda=0
lmda=0
layer=1234
lr=1e-2
wd=5e-3
mmt=0.9

DATASETS=(MIT_67 CUB_200_2011 Flower_102 stanford_40 stanford_dog)
DATASET_NAMES=(MIT67Data CUB200Data Flower102Data Stanford40Data SDog120Data)
DATASET_ABBRS=(mit67 cub200 flower102 stanford40 sdog120)

for i in 0 
do
    DATASET=${DATASETS[i]}
    DATASET_NAME=${DATASET_NAMES[i]}
    DATASET_ABBR=${DATASET_ABBRS[i]}

    CUDA_VISIBLE_DEVICES=0 \
    python -u train.py \
    --iterations ${iter} \
    --datapath data/${DATASET}/ \
    --dataset ${DATASET_NAME} \
    --name resnet18_${DATASET_ABBR}_reinit_lr${lr}_iter${iter}_feat${lmda}_wd${wd}_mmt${mmt}_${id} \
    --batch_size 64 \
    --feat_lmda ${lmda} \
    --lr ${lr} \
    --network resnet18 \
    --weight_decay ${wd} \
    --beta 1e-2 \
    --test_interval 1000 \
    --feat_layers ${layer} \
    --momentum ${mmt} \
    --reinit \
    --output_dir results/retrain \
    # &
    # --log \
    # &

done

[RudyChin]

Hi Ziqi,

For re-training, I use 90k instead of 30k iterations, maybe that can further improve your results from 44% to 59%. For fine-tuning, I'll update the scripts to include them later. But after a quick glimpse, the hyperparam for fine-tuning should be: lr 5e-3, wd1e-4, and mmt 0 the exact command is python -u train.py --iterations 30000 --datapath /data/MIT_67/ --dataset MIT67Data --name resnet18_mit67_lr5e-3_iter30000_feat0_wd1e-4_mmt0_1 --batch_size 64 --feat_lmda 0 --lr 5e-3 --network resnet18 --weight_decay 1e-4 --beta 1e-2 --test_interval 1000 --feat_layers 1234 --momentum 0 --log I just did a quick run and i can get 64.63% with the first 1k iteration for fine-tuning.

Hope this helps!

Rudy

[ziqi-zhang]

Hi Rudy,

Thanks very much for your quick reply. I checked the fine-tuning setting and it worked! I will do the retraining experiment again. Also, I was wondering is it possible to update the detailed setting for fine-tuning other datasets as I found this hyperparameter setting is very tricky and difficult to get. It will be very helpful if you can share the setting and others can reproduce the results.

Best, Ziqi

[RudyChin]

Hi Ziqi,

Yes, I agree. We've roughly performed a grid search over 8 configurations for each dataset as mentioned in our experimental setup. I'll put the script up later!

Thanks, Rudy