Minimal code for reproduce the results

[enkeejunior1]

Hi,

Thank you for your great work.

If you don't mind, could you provide us with minimal code or instructions to reproduce the results from the paper? Or, the minimal script to run the code would also be greatly appreciated.

Thank you in advance.

[tanganke]

Hi, @enkeejunior1 . The base models are downloaded from HuggingFace, and the datasets for image classification tasks are prepared according to the task arithmetic paper. You can refer to this issue for a detailed split definition for DTD, EuroSAT and SUN397.

1. Fine-tune CLIP models or flan-t5 models using three methods:
   • Full fine-tuning
   • LoRA fine-tuning
   • Linearized LoRA fine-tuning

Use the following scripts:

bash finetune_clip.sh

# for flan-t5 models
finetune_flan_t5.sh

2. Merge these fine-tuned models using different methods and evaluate their multi-task performance. Refer to the following scripts:
   • evaluate_multi_task_clip.sh
   • evaluate_multi_task_lm.py

This will iterate through all combinations of the downstream tasks, which is slow and computationally intensive. For more advanced merging methods, you can refer to FusionBench, where we provide instruction to run the experiments, the models and datasets will be automatically download from HuggingFace. However, unlike this method, the methods provided in FusionBench are currently focus on model fusion after the fine-tuning phase.

We also recommend to read:

[1] R. Jin, B. Hou, J. Xiao, W. Su, and L. Shen, “Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task Arithmetic.” arXiv, Jul. 09, 2024. doi: 10.48550/arXiv.2407.07089.

which also provide linear property of the fine-tuning process without introducing any inference cost.

[tanganke]

You can also just download the datasets from HuggingFace. However, the scripts to load dataset need to be modified. Here in this work, we still load datasets from raw image files.

The first two levels of the raw dataset directory:

○ → tree -L 2
.
├── dtd
│   ├── images
│   ├── imdb
│   ├── labels
│   ├── test
│   └── train
├── EuroSAT_splits
│   ├── test
│   ├── train
│   └── validation
├── gtsrb
│   ├── GT-final_test.csv
│   └── GTSRB
├── MNIST
│   └── raw
├── resisc45
│   ├── NWPU-RESISC45
│   ├── resisc45-test.txt
│   ├── resisc45-train.txt
│   └── resisc45-val.txt
├── stanford_cars
│   ├── cars_test
│   ├── cars_test_annos_withlabels.mat
│   ├── cars_train
│   └── devkit
├── sun397
│   ├── README.txt
│   ├── test
│   └── train
└── svhn
    ├── test_32x32.mat
    └── train_32x32.mat

[enkeejunior1]

Thank you so much for your comprehensive instructions and prompt reply! They're incredibly helpful.

I'm particularly grateful that you introduced me to Fusionbench. It appears to be very user-friendly, and I'm excited to incorporate it into my future research projects.

Your guidance is truly appreciated. Thanks again for your time and expertise!