search

JieShibo / PETL-ViT

[ICCV 2023] Binary Adapters, [AAAI 2023] FacT, [Tech report] Convpass
MIT License
162 stars 7 forks source link

Issue about training configuration on Binary Adapter with VTAB-1k Dataset #19

Open leoli646 opened 2 months ago

leoli646 commented 2 months ago

Hello, When I tried to replicate your binary_adapter experiment using the VTAB-1k dataset, I was unable to reproduce the results that you reported. I would like to discuss some potential issues with the training configuration that might be causing this discrepancy.

In similar works like VPT and SSF, different hyper-parameters (such as lr_rate, weight-decay, drop-path, etc.) are utilized for various datasets within VTAB-1k. However, the train.sh script in the binary_adapter codebase doesn't seem to account for these variations and applies default hyperparameters universally.

Could you advise on whether I should:

  1. Conduct a grid search to find the best hyperparameter set for each dataset?
  2. Or, should I use the hyperparameter settings from another public work like SSF, for instance?

Your insights would be greatly appreciated as I continue my experiments.

Looking forward to your reply!

JieShibo commented 2 months ago

In our experiments, we only searched for the scale factor. All the experiments are conducted on RTX3090 GPUs and may exhibit slight variations in results when executed on different devices. Further exploration of hyperparameters such as learning rate and weight decay could potentially enhance performance.