bash scripts for generalization experiments using task_arithmetic & ties-merging

[sooyonee]

Hi, I am interested in your unseen dataset experiment and trying to retrieve Table 4. I could retrieve accuracy for task arithmetic, however, I'm having difficulty retrieving the result for TIES-merging. Can you share your bash file and dataset setup for experiment in Table 4? Thanks!

[sooyonee]

Also, I would appreciate it if you let me know how to setup ood datasets (where to download, if necessary how to split). Thank you!

[tanganke]

The scripts to run it can be found at https://github.com/tanganke/weight-ensembling_MoE/tree/master/results

I have uploaded the ood datasets to HuggingFace. You can try to download and use it. You can reproduce the results for ties-merging using fusion_bench. However, the checkpoints used in FusionBench are fine-tuned by ourselves but in the paper, we use the checkpoints from task arithmetic. So the results would be different.

[sooyonee]

Thanks for your help. Now I am using fusion bench, and I've got another question. I'm using fusion bench to retrieve accuracies and experiment with my own algorithm, and I'm having difficulty running robustness experiments. For corrupted cases, it does not work with bash files given on the GitHub and project page. running following commands (original bash file without modelpool, taskpool specification did not work out so I modified), I got into the error. It seems that with the following command,

fusion_bench --config-name clip-vit-base-patch32_robustness_corrupted \
    corruption=$corruption \
    method=task_arithmetic \
        method.scaling_factor=0.3 \
    modelpool=CLIPVisionModelPool/clip-vit-base-patch32_robustness_corrupted \
    taskpool=clip-vit-base-patch32_robustness_corrupted \
    report_save_path=outputs/clip-vit-base-patch32_robustness_corrupted_${corruption}_task_arithmetic.json

error occurs as follows:

Error executing job with overrides: ['corruption=gaussian_noise', 'method=ties_merging', 'modelpool=CLIPVisionModelPool/clip-vit-base-patch32_robustness_corrupted', 'taskpool=clip-vit-base-patch32_robustness_corrupted']
Traceback (most recent call last):
  File "/data1/mulan98/anaconda3/envs/moe/lib/python3.10/site-packages/fusion_bench/scripts/cli.py", line 38, in main
    program.run()
omegaconf.errors.ConfigAttributeError: Missing key run
    full_key: run
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

with following warning [11/19/24 06:21:25] WARNING Unused argument: corruption=gaussian_noise It seems program: BaseHydraProgram = instantiate(cfg) in fusion_bench/scripts/cli.py is not properly instantiated as a program (FabricModelFusionProgram) but becomes Dictionary object (omegaconf.dictconfig.DictConfig). I tried to use your WebUI to generate configuration, but it did not support robustness experiment. I would appreciate it if you let me know I to tackle it. Thanks!

[tanganke]

I have updated the config file to fix this:

https://github.com/tanganke/fusion_bench/blob/23c1218fbaf5082aebe2ab693d0b1db00d6fcd36/config/clip-vit-base-patch32_robustness_corrupted.yaml#L11-L12

run git pull to update the code.

[sooyonee]

It helped thanks!

Additionally, is there any way to register new task vector obtained from new dataset? https://tall-masks.github.io includes 14, 20 datasets for CLIP experiment and I have their pre-trained weights and datasets. To include them in the taskpool and modelpool is it possible to do by modifying package source code? Or is this impossible since data and weights are located in huggingface server?

Thanks a lot for your help.

[tanganke]

We do not have a model pool class to handle open_clip models yet because it looks like that a specific version of open_clip package is required to load models from pickled binary checkpoint. But it is possible to merge open_clip vision models using FusionBench.

1. Import algorithm class in your own code:

# load open_clip vision models
pretrained_model = load_open_clip_model(...) # this loads an nn.Module object
# load task-specific models
task_a_model = load_open_clip_model(...)
task_b_model = load_open_clip_model(...)
...
task_n_model = load_open_clip_model(...)

You can import an algorithm class from FusionBench and merge:

from fusion_bench.method import TiesMergingAlgorithm

merged_model = TiesMergingAlgorithm(scaling_factor=... , threshold=..., ...).run({
  '_pretrained_': pretrained_model, 
  'task_a': task_a_model, 
  'task_b': task_b_model, 
  'task_n': task_n_model
})

2. Implement model pool and task pool class for openclip models.

You can refer to the implementation of CLIPVisionModelPool and CLIPVisionModelTaskPool. These are major methods that need to be implemented:

class YourModelPool:
    def load_model(self, ...):
        ...
    def load_processor(self, ...):
        ...
    def load_test_dataset(self, ...):
        # for test-time adaptation
        ...

class YourTaskPool:
    def evaluate(self, merged_model):
        ...