VL-PET

Official code for "VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control" (ICCV2023)

Authors: Zi-Yuan Hu^1,3, Yanyang Li¹, Michael R. Lyu¹ and Liwei Wang^*1,2 (^*Corresponding Author)

¹The Chinese University of Hong Kong
²Centre for Perceptual and Interactive Intelligence
³Shanghai AI Laboratory

Project page (with more details and fun fact of our logo): VL-PET

Abstract

As the model size of pre-trained language models (PLMs) grows rapidly, full fine-tuning becomes prohibitively expensive for model training and storage. In vision-and-language (VL), parameter-efficient tuning (PET) techniques are proposed to integrate modular modifications (e.g., Adapter and LoRA) into encoder-decoder PLMs. By tuning a small set of trainable parameters, these techniques perform on par with full fine-tuning. However, excessive modular modifications and neglecting the functionality gap between the encoders and decoders can lead to performance degradation, while existing PET techniques (e.g., VL-Adapter) overlook these critical issues.

In this paper, we propose a Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose effective control over modular modifications via a novel granularity-controlled mechanism. Considering different granularity-controlled matrices generated by this mechanism, a variety of model-agnostic VL-PET modules can be instantiated from our framework for better efficiency and effectiveness trade-offs. We further propose lightweight PET module designs to enhance VL alignment and modeling for the encoders and maintain text generation for the decoders.

Extensive experiments conducted on four image-text tasks and four video-text tasks demonstrate the efficiency, effectiveness and transferability of our VL-PET framework. In particular, our VL-PET-large with lightweight PET module designs significantly outperforms VL-Adapter by 2.92% (3.41%) and LoRA by 3.37% (7.03%) with BART-base (T5-base) on image-text tasks. Furthermore, we validate the enhanced effect of employing our VL-PET designs on existing PET techniques, enabling them to achieve significant performance improvements.

VL-PET Framework

Experiments

Quick Start

1. Installation

conda create -n vlpet
conda activate vlpet
pip install -r requirements.txt
python -c "import language_evaluation; language_evaluation.download('coco')"

2. Dataset Preparation

You are recommended to follow the dataset downloading instruction of VL-Adapter.

The following is the file structure of the datasets for your convenience:

3. Training & Evaluation (VL-PET-large)

Taking VL-PET-large as an example, we can conduct training and evaluation on different tasks as follows:

• VL-PET-large on image-text tasks (BART-base)

  # VL-PET-large on image-text tasks (BART-base)
  bash scripts/image-text/VL-PET-large.sh 20000 96 4 96 96 1e-3 42

  Click for more details...

  The content of scripts/image-text/VL-PET-large.sh: ```bash task=multitask model="bart" echo $model if [ $model == "t5" ] then folder_prefix="VLT5" backbone="t5-base" batch_size=300 elif [ $model == "bart" ] then folder_prefix="VLBart" backbone="facebook/bart-base" batch_size=500 fi echo $folder_prefix echo $backbone feature=RN101 lr=$6 sh=Encoder_MultiheadDownAdapter_dim$2_head$3_GatingLowRankLN_dim$4_Decoder_VPAdapter_dim$5_lr$6_seed$7 name=${sh}_${feature}__bs${batch_size}_image224_lr${lr} output=snap/${folder_prefix}_${task}/$name TOKENIZERS_PARALLELISM=True PYTHONPATH=$PYTHONPATH:./src \ python -m torch.distributed.launch \ --nproc_per_node=1 \ --master_port=$1 \ src/${task}.py \ --distributed --multiGPU \ --optim adamw \ --warmup_ratio 0.1 \ --clip_grad_norm 5 \ --lr ${lr} \ --epochs 20 \ --num_workers 4 \ --backbone ${backbone} \ --output $output \ --num_beams 5 \ --batch_size ${batch_size} \ --valid_batch_size ${batch_size} \ --reduction_factor 8 \ --use_tasks_prompts \ --tasks "vqa,gqa,nlvr,caption" \ --feature ${feature} --n_boxes 36 --downsample \ --image_size "(224,224)" \ --run_name $name \ --use_adapter \ --use_single_adapter \ --no_encoder_adapter \ --use_adapter_down_dim \ --use_encoder_adapter_down_multihead \ --adapter_down_dim $2 \ --encoder_adapter_multihead_num_head $3 \ --use_encoder_adapter_gating_large_x_lowrank \ --adapter_gating_down_dim $4 \ --unfreeze_encoder_layer_norms \ --no_decoder_adapter \ --use_decoder_enc_attn_value_parallel_adapter_down_dim \ --decoder_enc_attn_value_parallel_adapter_down_dim $5 \ --seed $7 ``` Since our code is built upon VL-Adapter, some arguments of VL-Adapter have been preserved for the convenience of conducting extensive experiments. For the arguments of the running command, you can refer to src/param.py. The following is the description of some selected arguments: ```bash backbone="facebook/bart-base" # use bart-base, hidden dimension d = 768 batch_size=500 # batch size feature=RN101 # visual features --lr ${lr} # learning rate --warmup_ratio 0.1 # warmup ratio --epochs 20 # training epochs --output $output # to store the results --use_tasks_prompts # use task prompts --tasks "vqa,gqa,nlvr,caption" # multi-task learning --seed $7 # use three different seeds, such as 42, 43 and 9595 # use shared-weight adapter-like modules --use_single_adapter # for encoder VL-PET module # encoders: r = 96, s = 1.0, N_h= 4 --no_encoder_adapter --use_adapter_down_dim --use_encoder_adapter_down_multihead --adapter_down_dim $2 --encoder_adapter_multihead_num_head $3 --use_encoder_adapter_gating_large_x_lowrank --adapter_gating_down_dim $4 --unfreeze_encoder_layer_norms # for decoder VL-PET module # decoders: r = 96, s = 1.0, N_h= 1 --no_decoder_adapter --use_decoder_enc_attn_value_parallel_adapter_down_dim --decoder_enc_attn_value_parallel_adapter_down_dim $5 ```

• VL-PET-large on image-text tasks (T5-base)

  # VL-PET-large on image-text tasks (T5-base)
  bash scripts/image-text/T5-VL-PET-large.sh 20001 192 4 192 0.3 96 3e-4 42

• VL-PET-large on video-text tasks (BART-base)

  # VL-PET-large on video-text tasks (BART-base)
  bash scripts/video-text/VL-PET-large.sh 20002 96 4 96 96 7e-4 20 42

Code Structure

The following is the file structure of VL-PET project for your convenience:

Running Command

For other experiments, we can replace VL-PET-large in the .sh file name with VL-PET-middleX, VL-PET-middleY, VL-PET-small, full_finetuning, bitfit and so on. The details of the hyper-parameters are reported in the appendix of our paper.

1. VL-PET-large

Please refer to Quick Start.

2. VL-PET-middleX

3. VL-PET-middleY

4. VL-PET-small

5. Baselines

Checkpoints & Logs

We provide checkpoints & logs for BART-base on image-text tasks as follows:

Acknowledgements

This work benefits from VL-Adapter, Ladder-Side-Tuning and unify-parameter-efficient-tuning. Our logo is borrowed from OpenMoji. Thanks for their awesome works!

Reference

If you find VL-PET useful for your research, please consider giving this repository a star and citing our paper as follows:

@inproceedings{hu2023vlpet,
  title     = {VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control},
  author    = {Zi-Yuan Hu, Yanyang Li, Michael R. Lyu and Liwei Wang},
  booktitle = {ICCV},
  year      = {2023}
}