Confused regarding which attention maps are stored and used for computing loss.

[HashmatShadab]

https://github.com/WindVChen/DiffAttack/blob/523c1af0d531c799565f8b258a540fda3c20c278/attentionControl.py#LL24C8-L26C65

Hi, Thanks for sharing your work! Can you please explain what's happening in the above lines of code. To my understanding, we are passing the 2[original latents, latents] and corresponding 2[unconditional embedding, conditional embedding] through the model. In the above lines of code why are we only taking half of the computed attetnion maps? Are we just storing the attention scores when we pass the conditional embedding?

[WindVChen]

Hi @HashmatShadab ,

Thanks for your attention to our work. It is true that we are passing 2×[original latent, modified latent], together with [2×unconditional embedding, 2×conditional embedding] into the model. Therefore, the attn here is of 4 batches, where the last two attention maps correspond to original latent and modified latent respectively, both under the guidance of conditional embedding.

We only leverage the last two batches for cross-attention and self-attention controls. For self-attention, you can find the control loss inside attentionControl.py, while for cross-attention, the control loss can be found inside diff_lantent_attack.py.

You might also consider applying additional control over self-attention in the first two batches. It will achieve better structure preservation, but also sacrifice some transferability.

[HashmatShadab]

Thank you for clarifying and for your suggestions.