While this alignment should not be applied to sqr's inference pipeline, we accidentally applied it. However, for (SQR)Deformable-DETR, the cls branches and box branches are shared across all levels. So, this bug only exists logically, but did not actually affect anything.
we found and fixed an unseen bug in SQR-Deformable-DETR:
The code logic of SQR-Deformable-DETR (and the original Deformable-DETR) is like:
calculate and update queries stage by stage in https://github.com/Fangyi-Chen/SQR/blob/ceb538f2a69e0c206a9e22e39ecdb75d392521d8/mmdet/models/utils/transformer.py#L707
collect the calculated queries.
feed the collected queries to the cls branch and box branch in https://github.com/Fangyi-Chen/SQR/blob/ceb538f2a69e0c206a9e22e39ecdb75d392521d8/mmdet/models/dense_heads/QR_deformable_detr_head.py#L17
for the original Deformable-DETR's training and inference, and for sqr's inference, we will have 6 queries and 6 lvl of corresponding cls branches and box branches. so the alignment is simply like https://github.com/Fangyi-Chen/SQR/blob/ceb538f2a69e0c206a9e22e39ecdb75d392521d8/mmdet/models/dense_heads/deformable_detr_head.py#L161
But for sqr training, we will have to align the collected queries (32 queries) with the corresponding 6 lvl of cls branch and box branch https://github.com/Fangyi-Chen/SQR/blob/ceb538f2a69e0c206a9e22e39ecdb75d392521d8/mmdet/models/dense_heads/QR_deformable_detr_head.py#L161
While this alignment should not be applied to sqr's inference pipeline, we accidentally applied it. However, for (SQR)Deformable-DETR, the cls branches and box branches are shared across all levels. So, this bug only exists logically, but did not actually affect anything.
We have added two lines of code to force SQR-Deformable-DETR to use the original pipeline, as in https://github.com/Fangyi-Chen/SQR/blob/ceb538f2a69e0c206a9e22e39ecdb75d392521d8/mmdet/models/dense_heads/QR_deformable_detr_head.py#L158