calculate iou

[onlyonewater]

The function calculate_IoU_batch in criteria.py, Why do you add 1 to the numerator and denominator when you compute the IOU? I am very confused!! It feels unfair!!

[liudaizong]

You can find the same calculation in CMIN(SIGIR19) https://github.com/ikuinen/CMIN_moment_retrieval/

[onlyonewater]

I know, but it feels unfair!!! It doesn't make any sense!!

[SCZwangxiao]

Maybe it's used to avoid zero division error. I met this problem when I was reproducing LGI (https://github.com/JonghwanMun/LGI4temporalgrounding). I added 1e-10 instead of 1 and the results didn't change.

[onlyonewater]

hi, @SCZwangxiao , when I use the code in (https://github.com/IsaacChanghau/VSLNet), I added 1 to the numerator and the denominator of the IOU, the results have changed a lot.

original results in charades-sta with finetune features: Rank@1, IoU=0.3: 69.89 Rank@1, IoU=0.5: 54.11 Rank@1, IoU=0.7: 34.35 mean IoU       : 49.55

add 1 to calculating iou: Rank@1, IoU=0.3: 72.77 Rank@1, IoU=0.5: 56.18 Rank@1, IoU=0.7: 37.55 mean IoU : 52.08

so, I think when calculate iou, 1 shouldn't add to the numerator and the denominator. That's wrong!!

[SCZwangxiao]

hi, @SCZwangxiao , when I use the code in (https://github.com/IsaacChanghau/VSLNet), I added 1 to the numerator and the denominator of the IOU, the results have changed a lot.

original results in charades-sta with finetune features: Rank@1, IoU=0.3: 69.89 Rank@1, IoU=0.5: 54.11 Rank@1, IoU=0.7: 34.35 mean IoU : 49.55

add 1 to calculating iou: Rank@1, IoU=0.3: 72.77 Rank@1, IoU=0.5: 56.18 Rank@1, IoU=0.7: 37.55 mean IoU : 52.08

so, I think when calculate iou, 1 shouldn't add to the numerator and the denominator. That's wrong!!

Oh! You are right. IoU will be definitely larger. I think '1' in the numerator should be removed, and that in the denominator should be changed into a very small number.

[onlyonewater]

yeah, I also think so.

[2o2o-2o2o]

@onlyonewater Hi, I have conducted experiments on the AcitivtyNet Captions dataset and here is the results:

1. Results in paper: R@1, IoU=0.3/0.5/0.7: 68.52/49.11/29.15 R@5, IoU=0.3/0.5/0.7: 87.68/77.43/59.63

2. Results based on the code in this repository: R@1, IoU=0.3/0.5/0.7: 68.48/49.21/29.12 R@5, IoU=0.3/0.5/0.7: 87.60/77.36/59.55

3. Results after modifying the calculation method of IoU: R@1, IoU=0.3/0.5/0.7: 67.68/48.57/28.79 R@5, IoU=0.3/0.5/0.7: 87.00/76.86/59.07

I think the results of IoU are slightly larger.

Details of the modification are as follows: I just rewrote some functions in criteria.py:

def compute_tiou(pred, gt):
    intersection = max(0, min(pred[1], gt[1]) - max(pred[0], gt[0]))
    union = max(pred[1], gt[1]) - min(pred[0], gt[0])
    return float(intersection) / union

def calculate_IoU(i0, i1):
    return compute_tiou(i0, i1)

def calculate_IoU_batch(i0, i1):
    all_tiou = np.ones(i0[0].shape)
    for i in range(len(i0[0].shape)):
        pred = [i1[0][i], i1[1][i]]
        gt = [i0[0][i], i1[1][i]]
        tiou = compute_tiou(pred, gt)
        all_tiou[i] = tiou
    return all_tiou

In addition, the model implemented by the code is quite different (e.g., only one layer Graph) from the model described in the paper. However, the results of the code are consistent with the paper. It feels so strange (You can see #3 ). Do you known the reason why?

[SCZwangxiao]

@onlyonewater Hi, I have conducted experiments on the AcitivtyNet Captions dataset and here is the results:

1.

Did you train ActivityNet model from scratch, or just used the pretrained model for evaluation? The code is a bit strange and I don't think it's the final version. I found that the TACoS dataloader is even imcompatible with the features! The dataloader is for seperate .npy files but the author gave us .h5py file.

[2o2o-2o2o]

@onlyonewater Hi, I have conducted experiments on the AcitivtyNet Captions dataset and here is the results:

Did you train ActivityNet model from scratch, or just used the pretrained model for evaluation? The code is a bit strange and I don't think it's the final version. I found that the TACoS dataloader is even imcompatible with the features! The dataloader is for seperate .npy files but the author gave us .h5py file.

Thanks for your quick reply. I just used the pretrained model for evaluation. In addition, the evaluation process is time-consuming. I have tried to retrain the model, but there have been some bugs that I can't fix.