How about not sharing point-level queries in the one2many assigner

[JingweiZhang12]

@LegendBC @wondervictor @outsidercsy Thanks for your code! I'm curious that sharing point-level queries could cause interaction between one2one group and one2many group in the training, which is not the desired operation in the original H-DETR. Have you tried both using isolated point queries and instance queries between one2one group and one2many group? BTW, Did the vanilla self-attention in the following ablation use the same setting as other self-attention variants?

[LegendBC]

Hi @JingweiZhang12 , thanks for your interest!

1. We have not tried isolated point queries for one2many group. One2one and one2many are both at the instance level. Since we think that the point-level queries do not provide leaked information from auxiliary instances. All the point-level interactions are performed intra-instance. We only apply the self-attn mask on inter-instance self-attn to prevent data leakage.
2. In TABLE 5, the vanilla self-attention experiment uses the same setting as others.