Questions about the activation collections.

[Pandint]

Thank you for this great work! The proposed visual task vectors are very interesting.

I am encountering some problems when running your codes and hope to get your help. In your paper, the task vectors are collected on 10 images. But in the released codes, the vectors are searched on the entire pascal datasets. Could you please clarify which one is correct?

Hope to get your response.

[alhojel]

Hey! Thanks for your interest in our project.

So when it comes to training REINFORCE to converge on the patching positions given a set of mean activations, we only use 10 examples.

This can be changed via the --train_images parameter in the train_reinforce file: https://github.com/alhojel/visual_task_vectors/blob/96e5e8302ca53dea054fb4abc5c943d7aaf46611/evaluate/reinforce_train.py#L36

However, if you're referring to how the mean activations are calculated, we only use 100 examples to calculate them and can be changed via the following parameter: https://github.com/alhojel/visual_task_vectors/blob/96e5e8302ca53dea054fb4abc5c943d7aaf46611/evaluate/collect_attention_heads.py#L28

I hope this clarifies some of your issues. Another thing to note is you can provide a list of query and support pairs so that your mean activations are calculated on a specific set of prompts: https://github.com/alhojel/visual_task_vectors/blob/96e5e8302ca53dea054fb4abc5c943d7aaf46611/evaluate/collect_attention_heads.py#L30

The utility of the query support pair list is so that you can select examples where the original model performed well hence increasing the quality of the mean activations.

Let me know if there is anything else I can help with!

[Pandint]

Thank you for your quick reply! It helps me a lot!