ZhangYuanhan-AI / NOAH

[TPAMI] Searching prompt modules for parameter-efficient transfer learning.
MIT License
218 stars 11 forks source link
deep-learning domain-generalization pre-trained-model prompt-tuning pytorch transfer-learning visual-prompting

Neural Prompt Search

Yuanhan ZhangKaiyang ZhouZiwei Liu
S-Lab, Nanyang Technological University

TL;DR

The idea is simple: we view existing parameter-efficient tuning modules, including [Adapter](https://arxiv.org/abs/1902.00751), [LoRA](https://arxiv.org/abs/2106.09685) and [VPT](https://arxiv.org/abs/2203.12119), as prompt modules and propose to search the optimal configuration via neural architecture search. Our approach is named **NOAH** (Neural prOmpt seArcH). ---

[arXiv][project page]

Updatas

[05/2022] arXiv paper has been released.

Environment Setup

conda create -n NOAH python=3.8
conda activate NOAH
pip install -r requirements.txt

Data Preparation

1. Visual Task Adaptation Benchmark (VTAB)

cd data/vtab-source
python get_vtab1k.py

2. Few-Shot and Domain Generation

Quick Start For NOAH

We use the VTAB experiments as examples.

1. Downloading the Pre-trained Model

Model Link
ViT B/16 link

2. Supernet Training

sh configs/NOAH/VTAB/supernet/slurm_train_vtab.sh PATH-TO-YOUR-PRETRAINED-MODEL

3. Subnet Search

sh configs/NOAH/VTAB/search/slurm_search_vtab.sh PARAMETERS-LIMITES

4. Subnet Retraining

sh configs/NOAH/VTAB/subnet/slurm_retrain_vtab.sh PATH-TO-YOUR-PRETRAINED-MODEL

We add the optimal subnet architecture of each dataset in the experiments/NOAH/subnet/VTAB.

5. Performance

fig1

Citation

If you use this code in your research, please kindly cite this work.

@misc{zhang2022neural,
      title={Neural Prompt Search}, 
      author={Yuanhan Zhang and Kaiyang Zhou and Ziwei Liu},
      year={2022},
      eprint={2206.04673},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknoledgments

Part of the code is borrowed from CoOp, AutoFormer, timm and mmcv.

Thanks to Chong Zhou (https://chongzhou96.github.io/) for the code of downloading the VTAB-1k.

[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FZhangYuanhan-AI%2FNOAH&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=visitors&edge_flat=false)](https://hits.seeyoufarm.com)