Open mohanen opened 5 years ago
Hi, Mohanen
Thank you for your contribution. I answer your questions.
We have heard about Disparity map post-filtering which we can use from OpenCV. https://docs.opencv.org/3.3.0/d3/d14/tutorial_ximgproc_disparity_filtering.html
We are going to apply this method to libSGM and evaluate it's ability.
We are willing to implement sub-pixel interpolation, maybe it will lead to some performance loss however.
Regards,
Hi, @atakagi-fixstars,
In your latest code, the pyramid-like noise mentioned above is almost gone but still some noise exist, I was using an older commit before.
I already tried the WLS filter.
I was using WLS with Confidence off and lambda and sigma values as preferred by OpenCV documentation.
Note: This same issue occurred for OpenCV SGBM (confidence on) too.
libSGM output
WLS applied
Hi, @mohanen
Thank you for your work! The WLS doesn't perform well... so we must find another way.
Hi @mohanen, I am having the same pyramid result as the one in your first comment. Could you please share the method for removing it?
Kind regards, Jiahao
If you are upscaling the disparity map means use nearest neighbor interpolation.
@mohanen Thank you for your reply! Could you please give me an example or a link of which code I could refer to?
For those who are wondering how to use OpenCV DisparityWLSFilter
:
The trick is that you would need the negate the right disparity when passing it to DisparityWLSFilter::filter
.
Below is an example:
// To calculate right disparaity, you need to flip the images and then flip the disparity back.
cv::Mat rightDisparity(right.size(), CV_16S);
cv::Mat flippedRightDisparity(right.size(), CV_16S);
cv::Mat flippedLeft(left.size(), CV_8U);
cv::Mat flippedRight(right.size(), CV_8U);
cv::flip(left, flippedLeft, 1);
cv::flip(right, flippedRight, 1);
sgm::StereoSGM ssgm_right(left.cols, left.rows, disp_size, input_depth, output_depth, sgm::EXECUTE_INOUT_HOST2HOST, param);
ssgm_right.execute(flippedRight.data, flippedLeft.data, flippedRightDisparity.data);
flip(flippedRightDisparity, rightDisparity, 1);
// OpenCV's DisparityWLSFilter implicitly assumes that the disparity is 16SC1 format
// and has been scaled up by 16 (yeah this part is very confusing).
// Therefore, for right disparity, you need to multiply by 16, and negate it (hence -16).
rightDisparity *= -16;
...
// Now you can call OpenCV's DisparityWLSFilter to give you a decent filtered disparity:
Ptr<DisparityWLSFilter> wlfFilter = createDisparityWLSFilterGeneric(true);
cv::Mat filteredDisp;
...
wlsFilter ->filter(disparity, left, filteredDisp, rightDisparity);
// OpenCV's DisparityWLSFilter implicitly assumes that the disparity is 16SC1 format // and has been scaled up by 16 (yeah this part is very confusing). // Therefore, for right disparity, you need to multiply by 16, and negate it (hence -16).
@lingjiankong The 16 is due to the fact that its a fixed point representation fixed<16,4> and requires conversion to float to be used.
fixed<16,4> denotes a 16-bit fixed-point number, of which 4 rightmost bits are fractional.
To convert that, perceive the bit string as an integer and divide it with 2 power number of bits used for the fraction part. 2^4 = 16.
@lingjiankong Thank you for your work!
If we shared param
with left matcher (or we shared sgm::StereoSGM instance as left and right matcher),
it is not needed to multiply 16
as sub pixel scale.
- rightDisparity *= -16;
+ rightDisparity *= -1;
disc | image |
---|---|
input | |
libSGM | |
libSGM(4PATH, w/ subpixel)+WLS | |
libSGM(4PATH, w/o subpixel)+WLS |
@sotsuka-fixstars Great results. The WLS filter looks surprisingly promising. Can you share any pointclouds of your version of WLS filter? Does it have the same pyramid effect that @mohanen has described in this thread?
@funmonty I thought so at a first glance. However, WLS Filter does not work fine in KITTI benchmark at our experience. @8BitCatJQW made another point cloud application using libSGM. It may help you.
https://github.com/8BitCatJQW/libsgmIssueDetails
I modified it, earned following outputs by using different filter.
filter | KITTI stereo 2015 000006_10 | Middlebury 2006 Baby |
---|---|---|
None | ||
WLS | ||
Bilateral |
@sotsuka-fixstars, Can you share the filter code snippets.
@mohanen Is a following snippet an answer to the request?
// WLS
cv::Ptr<cv::ximgproc::DisparityWLSFilter> wls_f = cv::ximgproc::createDisparityWLSFilterGeneric(true);
cv::Mat wls_filtered_disp;
wls_f->filter(disparity, I1, wls_filtered_disp, disp_right);
// Bilateral
cv::cuda::GpuMat g_disp, g_filtered_disp;
g_disp.upload(disparity);
cv::cuda::bilateralFilter(g_disp, g_filtered_disp, kernel_size, sigma_color, sigma_spatial);
cv::Mat bilateral_filtered_disp;
g_filtered_disp.download(bilateral_filtered_disp);
kernel_size
, sigma_color
, sigma_spatial
come from command line argument.
I decided these params slackly because I didn't know proper values of these...
is this implementation the cuda version of opencv StereoSGBM? i found the result of the two is different. there is a blocksize parameter in StereoSGBM, which i cann't found here. thanks for the explanation!
Hi, @ynma-hanvo
is this implementation the cuda version of opencv StereoSGBM?
No.
In libSGM, 9x7 center-symmetric census transform is used for matching cost.
@mohanen Is a following snippet an answer to the request?
// WLS cv::Ptr<cv::ximgproc::DisparityWLSFilter> wls_f = cv::ximgproc::createDisparityWLSFilterGeneric(true); cv::Mat wls_filtered_disp; wls_f->filter(disparity, I1, wls_filtered_disp, disp_right); // Bilateral cv::cuda::GpuMat g_disp, g_filtered_disp; g_disp.upload(disparity); cv::cuda::bilateralFilter(g_disp, g_filtered_disp, kernel_size, sigma_color, sigma_spatial); cv::Mat bilateral_filtered_disp; g_filtered_disp.download(bilateral_filtered_disp);
kernel_size
,sigma_color
,sigma_spatial
come from command line argument. I decided these params slackly because I didn't know proper values of these...
Does the bilateral filter work in subpixel mode? Do we need to filter off the invalid pixels first for better or accurate filtering?
The Disparity from libSGM, when projected in a point cloud the output contains noise like a pyramid in front of the camera covering the entire scene or the edges of the of the object seem to replicate multiple times in various disparity levels.
I have some knowledge on how sgm works and for post-processing I have been looking into Nerian's post-processing Steps and Hirshmullers Disparity Refinement Steps to improve the disparity map.
I was able to remove the noise (mentioned above) using a speckle filter which retained the correct data but at the cost of making it sparse almost losing more than 50% of the data in some scenes.
And looking at the other post-processing steps like Uniqueness check, Consistency check, Filtering of untextured image areas, Noise reduction seems to make the disparity even sparser and applying an optimal hole/gap filling technique wouldn't be enough I think.
I know SGM can do it better it even won 11th place in the ROB 2018.
So do you guys have any idea on
Thanks and Regards, Mohanen B