Bug: oLRP

[priteshgohil]

So I was testing LRP code with just GT and prediction of one image. First five categories have no prediction or ground truth labels and therefore I get -1 value which makes sense. However, in 2. Mean Optimal LRP and Components: simply average is taken over all the value resulting in negative mean optimal values.

Is it okay to have negative values? Ideally we should not consider those classes in mean calculation which doesn't exist. So I guess it should be sum of three class divide by three. (Correct me if I'm wrong)

Following are the result of my LRP calculation.

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
DONE (t=0.00s).
Accumulating evaluation results...
/home/pgohil/LRP/cocoLRPapi-master/PythonAPI/pycocotools/cocoevalLRP.py:336: RuntimeWarning: invalid value encountered in double_scalars
  LocError[s,k]=np.sum(IoUoverlap[:thrind])/omega[s,k];
/home/pgohil/LRP/cocoLRPapi-master/PythonAPI/pycocotools/cocoevalLRP.py:337: RuntimeWarning: invalid value encountered in double_scalars
  FPError[s,k]=nhat[s,k]/(omega[s,k]+nhat[s,k]);
DONE (t=0.01s).
oLRP, moLRP and Class Specific Optimal Thresholds are as follows: 

------------------------------------------------------ 

------------------------------------------------------ 

1.Optimal LRP and Components:
------------------------------------------------------ 

oLRP=[[-1.         -1.         -1.         -1.         -1.          1.     0.94463622       1. ]]

oLRPLocalization=
[[-1.         -1.         -1.         -1.         -1.          nan    0.29930628         nan]]

oLRPFalsePositive=
[[-1.   -1.   -1.   -1.   -1.     nan  0.85   nan]]

oLRPFalseNegative=
[[-1.         -1.         -1.         -1.         -1.          1.    0.36842105        1. ]]

------------------------------------------------------ 

------------------------------------------------------ 

2.Mean Optimal LRP and Components:
------------------------------------------------------ 

moLRP=-0.2569, moLRP_LocComp=-0.7834, moLRP_FPComp=-0.6917, moLRP_FPComp=-0.3289 

------------------------------------------------------ 

------------------------------------------------------ 

3.Optimal Class Specific Thresholds:

[[-1.   -1.   -1.   -1.   -1.    0.    0.08  0.  ]]
------------------------------------------------------ 

------------------------------------------------------ 

[kemaloksuz]

Hi,

Firstly, all the LRP values should be between 0 and 1, so you should get no negative results.

The reason why you get such values is your dataset normally consists of 8 classes as I see, however, you evaluate only 3 classes when you provide a single image as an input. And if you have a single image in which all the gt classes are not represented and at the same time the detector does not yield detections for these unrepresented classes, then obviously both gt and detection sets are empty, and there is nothing to evaluate. Unless both the gt set and the detection set are not empty, the LRP, and oLRP, are defined (not the components but the total error) and will be between 0 and 1.

In any case this was a bug in the code, and now I solved it. You should see nan values also for the classes without gt and detections now. Also, only the represented classes (either by a detection or by a gt) will be averaged over for the moLRP and its components, so in your example, you should see (1+0.94+1)/3=0.9815 for moLRP. After you check that you have 0.9815, please close this issue, else please let me know.

I also want to note that a new version of the repo will be released in short (hopefully in a month), and we will also consider this in the new version. Thanks for your contribution.

[priteshgohil]

Thank you for your quick response. I will update soon.

Update: @kemaloksuz Thank you again for fixing above issue. I have another quick question. (Attached data.zip can be used to reproduce the issue)

Results are as expected with GT.json and results.json. But when I reduce number of results to 21 from 100 and run with lessResults.json, I have the following error.

Traceback (most recent call last):
  File "evalDemoLRP.py", line 22, in <module>
    cocoEvalLRP.accumulate()
  File "/home/pgohil/LRP/cocoLRPapi-master/PythonAPI/pycocotools/cocoevalLRP.py", line 315, in accumulate
    for i in range(len(IoUoverlap)):

When I debug for IoUoverlap value, it was 0.0

So what could be the reason for this error?

[priteshgohil]

Also, can you also consider updating print statement in code? The last moLRP_FPCompshould be renamed to moLRP_FNComp

2.Mean Optimal LRP and Components:
------------------------------------------------------ 

moLRP=-0.2569, moLRP_LocComp=-0.7834, moLRP_FPComp=-0.6917, moLRP_FPComp=-0.3289 

[kemaloksuz]

Hello, both issues are corrected. I tried with your inputs and seems working.

Thanks for your correction about the print statement.