Question: Federated Affinity Fusion and `NewFusion`

[ConnorBaker]

Hello!

I had a few questions about the Federated Affinity Fusion module: https://github.com/yjsunnn/FBANet/blob/1b71421e4d303c9847dfe2dd6211453b0ceacb9c/model.py#L766-L888.

In the paper, it's described as differences of affinity maps: Section 4.3.

I'm not very familiar with many of these concepts, and I'm struggling to understand the code, in part because I can't map the names in the code to concepts in the paper.

Would you mind describing in a little bit more detail what is happening in NewFusion and how it relates to the Federated Affinity Fusion module described in the paper?

Thank you for sharing this code! Reading though it has been a wonderful learning experience!

[ConnorBaker]

Apologies if this wasn't the right place to post this -- let me know if you'd prefer emails for additional questions!

[yjsunnn]

Thanks for your interest in our job. The code of class NewFusion corresponds to the FAF in Section4.3. And the core idea of FAF is illustrated in Figure below.

So in the code, we first compute the similarity maps among each frame with the base frame (i.e., corr in line 845, it represents the similarity, or as the paper says, affinity among each frame with the base/reference frame). And then, we compute the difference map of affinity maps (i.e., corr_difference in line851) which corresponds to the d(A1,i, A1,1) in the equation (2). The difference maps are expected to pay attention to complementary details that do not appear in the base frame. And for the last step, we use the difference maps as the weight to accomplish the equation (as shown in line861-866, and the A1,1 is defaulted as 1).

Hope this explanation helps.

[yjsunnn]

By the way, I have updated my email in the Readme.md. Please feel free to contact me :)