problem or bug in refine_match

[pengfeidip]

Problem

In refine_match function, which try to ensure every groundtruth boxe can be assigned to a prior boxe(following code). But there is a problem , if there are same numbers inbest_prior_idx, it means not every groundtruth box will be assigned to a prior box.Actually, I think the problem is from the code of SSD match https://github.com/luuuyi/RefineDet.PyTorch/blob/0e4b24ce07245fcb8c48292326a731729cc5746a/layers/box_utils.py#L146-L151

---

An example

Assume there are 3 groundtruth boxes and 4 priori boxes, so you get a IoU matrix named overlap with size [3, 4] by function jacarrd

overlap = torch.rand(3, 4)
print(overlap)
# tensor([[0.5898, 0.4675, 0.0536, 0.2099],
#           [0.4566, 0.2781, 0.0106, 0.4372],
#           [0.5378, 0.8868, 0.2474, 0.1279]])

# best prior box for each gt box
best_prior_overlap, best_prior_idx = overlap.max(1, keepdim=False)
# best ground truth for each prior with shape [num_priors]
best_truth_overlap, best_truth_idx = overlap.max(0, keepdim=False)

print(best_truth_overlap)
print(best_prior_idx)
# tensor([0.5898, 0.8868, 0.2474, 0.4372])
# tensor([0, 0, 1])

best_truth_overlap.index_fill_(0, best_prior_idx, 2)
# tensor([2.0000, 2.0000, 0.2474, 0.4372])

Then, the following code cannot ensure that every groundtruth box is assigned to a prior box

for j in range(best_prior_idx.size(0)):
    best_truth_idx[best_prior_idx[j]] = j
print(best_truth_idx)
# tensor([1, 2, 2, 1])

In this case, there are 4 prior boxes, but only gt boxes with index 1 and 2 are assigned to priors. 0th gt box is not asssigned to any prior box...

If there are something wrong, please tell me, Thank you !!

[pengfeidip]

Finally, I don't think it's a big problem.... because there are so many prior boxes, thus it's almost impossible to have a GT box which has no corresponding prior box

[pengfeidip]

And if I'm right, then I have a solution for this problem:

Origin

 best_truth_overlap.index_fill_(0, best_prior_idx, 2)  # ensure best prior 
 # ensure every gt matches with its prior of max overlap 
 for j in range(best_prior_idx.size(0)): 
     best_truth_idx[best_prior_idx[j]] = j 
 matches = truths[best_truth_idx]     

Modified

 best_truth_overlap.index_fill_(0, best_prior_idx, 2)  # ensure best prior 

# get indexes of all GT boxes which are assigned to priors
truths_assigned = tensor.unique(best_truth_idx) 

# if j-th GT box is not assigned, then force the GT box to be matched with
# a prior which has highest IoU with it.  
for j in range(best_prior_idx.size(0)): 
    if j not in truths_assigned: 
        best_truth_idx[best_prior_idx[j]] = j 

 matches = truths[best_truth_idx]