search

yjsunnn / FBANet

Official implementation of ICCV2023 "Towards Real-World Burst Image Super-Resolution: Benchmark and Method"
MIT License
117 stars 7 forks source link

Question: Federated Affinity Fusion and `NewFusion` #6

Closed ConnorBaker closed 1 year ago

ConnorBaker commented 1 year ago

Hello!

I had a few questions about the Federated Affinity Fusion module: https://github.com/yjsunnn/FBANet/blob/1b71421e4d303c9847dfe2dd6211453b0ceacb9c/model.py#L766-L888.

In the paper, it's described as differences of affinity maps: Section 4.3.

I'm not very familiar with many of these concepts, and I'm struggling to understand the code, in part because I can't map the names in the code to concepts in the paper.

Would you mind describing in a little bit more detail what is happening in NewFusion and how it relates to the Federated Affinity Fusion module described in the paper?

Thank you for sharing this code! Reading though it has been a wonderful learning experience!

ConnorBaker commented 1 year ago

Apologies if this wasn't the right place to post this -- let me know if you'd prefer emails for additional questions!

yjsunnn commented 1 year ago

Thanks for your interest in our job. The code of class NewFusion corresponds to the FAF in Section4.3. And the core idea of FAF is illustrated in Figure below. image

So in the code, we first compute the similarity maps among each frame with the base frame (i.e., corr in line 845, it represents the similarity, or as the paper says, affinity among each frame with the base/reference frame). And then, we compute the difference map of affinity maps (i.e., corr_difference in line851) which corresponds to the d(A1,i, A1,1) in the equation (2). The difference maps are expected to pay attention to complementary details that do not appear in the base frame. And for the last step, we use the difference maps as the weight to accomplish the equation (as shown in line861-866, and the A1,1 is defaulted as 1).

Hope this explanation helps.

yjsunnn commented 1 year ago

By the way, I have updated my email in the Readme.md. Please feel free to contact me :)