Error for this code

[Diksha-Moolchandani]

What is the error for this code?

When I used the Middlebury evaluation SDK, I got a very high error of 85% if I set the bad threshold to 10. The AD-Census algorithm is run with the default disparity values. The final image generated is compared with the ground truth image present in the repo.

Upon increasing the threshold, the error decreases. However, this does not seem to be right. Because increasing the threshold means we are allowing a larger margin for error in the pixel values of the generated disparity and the ground truth disparity.

Is there some bug in the code or the method of evaluation needs to be changed? Can anyone help in this regard?

[ethan-li-coding]

1. the real disparity values are saved in array "disparity", which was defined as float32 type.
2. the final saved image who's format is "png" is for displaying, can't be used to evaluate.
3. you should save array "disparity" in PFM format, which can be done by Middlebury evaluation SDK, then evaluating with PFM file.
4. Besides, all valid values in array "disparity" should be multiply by a factor, because the ground-truth did the same thing when generating. Please visit the dataset site to get the factor value : https://vision.middlebury.edu/stereo/data/scenes2003/. For instance, the facor is 4 for 2003 datasets.

I'm sorry I didn't write this part of the code, considering that this will introduce other third-party libraries.

[Diksha-Moolchandani]

Thanks a lot for the reply.

However, this is specific to Middlebury. I want to evaluate it for KITTI 2015 training dataset: http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo For evaluating for KITTI, I used the disparity array and calculated the 3-pixel error with respect to the ground truth image that is in png format that is read using cv::imread. Also, there is no factor for the KITTI dataset.

The error comes out to be 67.6% for adcensus algorithm with min disparity 0 and max disparity 64. Can you help me with this dataset?

[ethan-li-coding]

Strange. Did you skip those invalid disparities which are equal to Invalid_Float? or did you only evaluate those pixels who have truth-value?

[Diksha-Moolchandani]

Yes. I used only valid values and those pixels that have positive value in the ground truth.

I used this code snippet:

double sum = 0; int count = 0; float ans = 0; cv::Mat myMap = cv::imread(gt_path);

    for (sint32 i = 0; i < height; i ++)
    {
            for (sint32 j = 0; j < width; j ++)
            {
                    if(myMap.at<uchar>(i,j) > 0 && abs(disparity[i * width + j]) != Invalid_Float ){
                            sum++;
                            ans = abs(disparity[i * width + j] - myMap.at<uchar>(i,j));
                            if ( (ans > 3.0) &&  (ans / myMap.at<uchar>(i,j)  > 0.05))
                                    count ++;
                    }
            }
    }
    double result = (count + 0.0)/sum;
    cout << "disp path is: " << gt_path << endl;
    cout << "error is: " << result * 100 << endl;

[ethan-li-coding]

Try to use cv::imread(gt_path,0)？ cv::imread load image in 3 channels by default.

[Diksha-Moolchandani]

Thanks a lot. That reduced the error to 18% for adcensus (min disparity 0 and max disparity 64).

[ethan-li-coding]

OK. Try to set “ad_option.do_filling = false” and then computing error again.

[Diksha-Moolchandani]

It reduced further to 10.99%

[ethan-li-coding]

OK. That's all I can think of for now.

[Diksha-Moolchandani]

thanks a lot.

[Diksha-Moolchandani]

With the above two changes, I am getting an error of 2.5% for semiglobal matching. The filling line was already false in semiglobal code. sgm_option.is_fill_holes = false;

[ethan-li-coding]

Sounds good

[Diksha-Moolchandani]

For PatchMatch, it is 14-15%. Can the error for PatchMatch and AD-Census be reduced further? Or are they inherently inferior to semi-global matching?