About reproduction.

[AtsuMiyai]

Hello. Thanks for the interesting work.

I have trouble with reproduction (I think this is a problem for me, not for you. ).

The reproduction result of LoRA on Caltech101 seems to be lower than the result in the paper.

We executed the following commands (without srun).

CONFIG=experiments/LoRA/ViT-B_prompt_lora_8.yaml
CKPT=ckpt/ViT-B_16.npz

for LR in 0.001
do
    for DATASET in caltech101
    do 
        python supernet_train_prompt.py --data-path=./vtab-1k/${DATASET} --data-set=${DATASET} --cfg=${CONFIG} --resume=${CKPT} --output_dir=./saves/${DATASET}_lr-${LR}_wd-${WEIGHT_DECAY}_lora_100ep_noaug_xavier_dp01_same-transform_nomixup --batch-size=64 --lr=${LR} --epochs=100 --is_LoRA --weight-decay=${WEIGHT_DECAY} --no_aug --mixup=0 --cutmix=0 --direct_resize --smoothing=0 
    done
done

With three different seeds, any accuracy did not exceed 90%. The output file is here.

The reproduction result of Adapter is significantly low especially on SVHN.

We executed the following commands (without srun).

CONFIG=experiments/Adapter/ViT-B_prompt_adapter_8.yaml
CKPT=ckpt/ViT-B_16.npz

for LR in 0.001
do
    for DATASET in svhn
    do 
        python supernet_train_prompt.py --data-path=./vtab-1k/${DATASET} --data-set=${DATASET} --cfg=${CONFIG} --resume=${CKPT} --output_dir=./saves/${DATASET}_${LR}_${WEIGHT_DECAY}_vpt --batch-size=64 --lr=${LR} --epochs=100 --is_visual_prompt_tuning --weight-decay=${WEIGHT_DECAY} --mixup=0 --cutmix=0 --drop_rate_prompt=0.1 --no_aug --inception --direct_resize --smoothing=0 
    done
done

The accuracy is around 35%. The output file is here.

I would appreciate any advice you can give me, no matter what information you have.🙇‍♂️

[ZhangYuanhan-AI]

Hi, thanks for your interest!

1. the first suggestion that I might want to give is : for each dataset, I usually run three times for three seeds, then report their average performance.

[AtsuMiyai]

Thanks for the quick reply.

I ran 5 times for 5 seeds. But, the results are 88.64, 88.91, 88.64, 88.10, 89.32.

Since the result in https://github.com/ZhangYuanhan-AI/NOAH/issues/15#issuecomment-1521458666 achieves more than 90% accuracy, is this due to differences in the experimental environment or hardware?

We can reproduce the results on other datasets with LoRA.

[MouxiaoHuang]

@AtsuMiyai I have reviewed the logs you provided, and here are some observations for your consideration:

1. Regarding the experiment "caltech101-lora," there's a warning that reads: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please use the InterpolationMode enum. While I'm uncertain if this warning is the root cause of the issue, it might be worth addressing to see if it helps.

2. Concerning the experiment "svhn-adapter," it appears that the hyper-parameters is_adapter=False and is_visual_prompt_tuning=True should be adjusted to is_adapter=True and is_visual_prompt_tuning=False. This adjustment seems to affect the adapter_dim in the sampled model configuration, resulting in a value of 0.

I hope these suggestions prove useful in resolving the issue.

[AtsuMiyai]

@MouxiaoHuang Thanks for your kind suggestion! I really appreciate it.

1. I modified the corresponding part and reimplemented it. But, the result was not changed. As you advised, by comparing my log with yours, I try to look for other causes.

2. Adapter works well. Thanks!