Possible bug ?

[viniciusarruda]

It seems that duplicated detections is currently discarted and not marked as false positive as should be.

The code extracted from here:

    if iouMax >= IOUThreshold:
        if det[dects[d][0]][jmax] == 0:
            TP[d] = 1  # count as true positive
            # print("TP")
        det[dects[d][0]][jmax] = 1  # flag as already 'seen'
    # - A detected "cat" is overlaped with a GT "cat" with IOU >= IOUThreshold.
    else:
        FP[d] = 1  # count as false positive
        # print("FP")

Should be:

    if iouMax >= IOUThreshold:
        if det[dects[d][0]][jmax] == 0:
            TP[d] = 1  # count as true positive
            # print("TP")
        else:                                              ## ADDED
            FP[d] = 1  # count as false positive           ## ADDED
        det[dects[d][0]][jmax] = 1  # flag as already 'seen'
    # - A detected "cat" is overlaped with a GT "cat" with IOU >= IOUThreshold.
    else:
        FP[d] = 1  # count as false positive
        # print("FP")

As the original code*:

    % assign detection as true positive/don't care/false positive
    if ovmax>=VOCopts.minoverlap
        if ~gt(i).diff(jmax)
            if ~gt(i).det(jmax)
                tp(d)=1;            % true positive
        gt(i).det(jmax)=true;
            else       %% THIS SHOULD BE ADDED
                fp(d)=1;            % false positive (multiple detection) 
            end
        end
    else
        fp(d)=1;                    % false positive
    end

*Download it here and take a look at line 93 from file VOCevaldet.m inside the folder VOCcode

Or am I missing something in your code that justifies it ? Thank you.

[ruslanasa]

Hi, all,

Any update on this issue?

[rafaelpadilla]

Dear @viniciusarruda,

Thanks for your feedback. Sorry it took me some days to answer, but I needed to perform some tests before getting back to you.

Yes, you are right! The code is a little different than the official implementation, but the difference in the Average Precision is almost undetectable and are not present in every case. The example shown in the README, for instance, is not affected by your suggestion. I believe this is the reason why I did not notice it easily before.

I ran the official PASCAL VOC Matlab code and compared with our results. The results to be compared are presented in the columns AP (our code before) and AP (official matlab code) on table below:

Observations:

• The official Matlab code considers only the ground truths tagged as 'not difficult'. Our code considers all cases included in the '\groundtruth' and 'detections' folders, because we leave to the user the decision to evaluate any desired bounding box. That's why, in this, test I included both 'difficult' and 'not difficult' bounding boxes. In case you want to reproduce the results, I can provide you the code if you want.

• Notice that only for two classes (cat and dog) the Average Precision changed. And it was only in the last decimal places.

• I considered your suggestion and changed the code. The results are now identical, including all decimal places (see the column AP (our updated code)). Please, use git pull command or clone the repository to obtain the changes.

I kindly ask you to close this issue if that resolves the question.

Thank you again.

[viniciusarruda]

Thank you !