search

PeizeSun / SparseR-CNN

[CVPR2021, PAMI2023] End-to-End Object Detection with Learnable Proposal
MIT License
1.31k stars 187 forks source link

Confused by the interaction between ROI features and proposal features #38

Open YueLiao opened 3 years ago

YueLiao commented 3 years ago

Thx for your interesting work and the code. I am confused by some points as follow when fusing ROI feats and proposal feats:

image

In this setting, the transformed proposal feature serves as a key, and the transformed ROI feature is regarded as a query. Firstly, producing an attention weight between ROI feature and proposal feature. Then, combining the attention weight with the other transformed proposal feature, which is a value. In this case, none of the appearance features (ROI) are kept, the interacted and output feature is a re-weighted proposal feature. The ROI feature only plays a role to compute an attention map. I do not understand why it works, predicting bounding-box without the appearance features. Additionally, it also looks different from the original paper: image And it is inversed to the multi-head attention in Detr. @PeizeSun

PeizeSun commented 3 years ago

Hi~ In fact, our dynamic interaction is not same as multi-head attention. We do not recommend you understand dynamic interaction by the concepts of query, key and value.

YueLiao commented 3 years ago

Hi~ In fact, our dynamic interaction is not same as multi-head attention. We do not recommend you understand dynamic interaction by the concepts of query, key and value.

Thx for your reply. I understand your mean where the dynamic filter is similar to what in the Siamese FC for single object tracking, where the proposal feature serves as a convolutional kernel and the ROI feature is a template. Moreover, I still have a question about ablation studies. Could you provide more details about the multi-head attention setting in Table 8, e.g., query, key, and value?

BreezeHavana commented 3 years ago

In fact, our dynamic interaction is not same as multi-head atten

Hi, thank you for your novel work and replies. I noticed that parameters was separated into two parts and interact with RoI separately, does it have special significance? thx in advance!

perrychen0413 commented 1 year ago

Hi, thanks for your work. I have the same question as BreezeHavana that why parameters was separated into two parts and interact with RoI separately? Has the question been answered?

BreezeHavana commented 1 year ago

auto reply