Filtering noise from depth image/point cloud

[mirzashah]

The point clouds generated by the RealSense have a bit of noise in them which appear as these sort of "ghost points" or "speckles" in mid-air. These ghosts points are problematic in places like the ROS nav stack in which they are marked as obstacles. I've tested this across 5 different sensors, so I'm presuming it's just a property of the sensor.

In order to use the RealSense's in the ROS nav stack, the point clouds have to be filtered. I've had very good luck removing these ghost points using PCL's statistical outlier filter through the StatisticalOutlierNodelet provided with pcl_ros.

However, this filter is extremely computationally intensive -- using one filter on the realsense running at 320x240,30 fps generates a filtered stream at 5 fps that uses 100% CPU on a dual core i7.

I've been able to reduce this computational load by throttling the depth image topic, binning the depth image (via image_proc/CropDecimate nodelet), projecting a new point cloud from it, and filtering that point cloud. This reduces the load and the sensors are now usable, but the load is still quite high due to the stat outlier filter. I'm still searching for better solutions.

My request is - can we get out-of-the-box filtering with the RealSense on the full resolution depth image/point cloud that is computationally efficient? This would make the RealSenses much more plug & play and much more useful for ROS applications.

• Has anybody come up with a better way to filter such as run a filter directly on the depth image or perhaps there are settings on the camera that can natively do this?
• Is it possible to do this filtering onboard the firmware of the sensor?
• If filtering can't be done on the hardware, could Intel or somebody else provide a set of nodelets that can do the filtering in software + launch files for them?

[reaganlo]

@mirzashah The point cloud with "ghost points", are you referring to -- the one that is generated by the RealSense nodelet itself OR -- the one that is generated by the "depth_image_proc/point_cloud_xyzrgb" nodelet in rgbd_launch using the native depth and color streams from the RealSense nodelet.

Just wanted to confirm.

[mirzashah]

@reaganlo The ghost points appear in both -- the default topics are /camera/depth/image (what the RealSense nodelet produces) and /camera/depth/points (which depth_image_proc/point_cloud_xyzrgb produces).

You can compare them in rviz by using the DepthCloud and PointCloud displays respectively. To me, this is an issue in the depth image generated by the RealSense. For example, if you point the sensor at the ground, it clearly sees it and generates a nice point cloud covering the ground. However, it also generates these ghost points between the floor and the camera and well below the floor as well particularly as you get further from the camera.

The depth image needs to be cleaned, one approach that comes to mind is doing a nearest neighbor search on the depth image and removing cells that are not at similar depths to neighboring cells or in isolation. I couldn't find any out-of-the-box filters that work directly on depth image sin ROS.

Just confirming, do you also see this phenomenon? If you have no idea what I'm talking about, I'll take a screen capture video and send you a link here. It's the middle of the night here, will take the capture tomorrow. I can show you the difference between a clean point cloud generated by the statistical outlier filter and the one the RealSense is generating.

Note that I'm using the latest firmware and using everything stock out-of-the-box with your driver. This phenomenon happens at 640x480 as well as 320x240. I tried playing with the controls you guys exposed with dynamic_reconfigure (e.g. camera gains) to see if those things could clean the noise -- no luck.

[ghindman]

@mirzashah we have an open investigation task for this. Stay tuned.

[mirzashah]

@ghindman Awesome thanks! We've been working on it ourselves...one of our engineers @safijari has had luck with an OpenCV median filter directly on the depth image. Computationally much more efficient but we're not able to remove all noise yet.

Also fyi - if you look at the depth image in greyscale in image_view, anywhere you see these white speckles flash -- those are the noise.

[safijari]

Just an update on @mirzashah's application: Some of the noise appears to be salt and pepper noise. A median filter on the depth image takes care of most of that. There is still other noise present that appears to be near the edges/transitions between two depth levels. For our application we were able to do some voxel based downsampling using PCL and remove any voxels that contain less than 15 points (that number is highly empirical). If someone needs to efficiently clean a point cloud (albeit at low resolutions) we can share how that's accomplished.

[mirzashah]

@ghindman @reaganlo @mdhorn

Here is a video showing you the difference between the filtered and unfiltered point clouds on our robot -- hope it makes the problem clearer.

https://www.dropbox.com/s/47k1rbrvfie5gpd/realsense_filter.mp4?dl=0

The video shows a "downsample depth image -> median filter depth image -> project point cloud -> voxel filter point cloud" chain approach. Note that this approach works for our use case of robot navigation and is computationally efficient -- however it comes at the cost of losing a lot of detail in the point cloud. Somebody doing 3d mapping, 3d object recognition, gesture recognition, or anything that requires more detailed point clouds will not be happy with this approach. I wouldn't close this issue until a better approach is found.

[aravindpreshant]

@safijari : could you please share how you cleaned up the point cloud?

[safijari]

Hey @aravindpreshant. As I described on my earlier post, I first applied a median filter to the depth image, converted that to a point cloud and then applied the VoxelGrid filter form PCL 1.7.2 for which the setMinimumPointsNumberPerVoxel(...) was set to some number that made sense for our cloud (30 I think).

There are two things to note here: 1) The resulting cloud is very low resolution since the VoxelGrid filter only keeps Voxel centroids* and 2) All of this was done in ROS (so the depth image to point cloud conversion came from a nodelet that comes with the PCL ROS package).

Are you using ROS? If yes then I can share some code snippets that should help you get a better cloud.

*Note that a) you don't have to use the VoxelGrid step since the median filter takes care of much of the noise and b) it should be possible to implement your own similar filter that removes the points associated with sparsely populated voxels and keeps the remaining points. Note that the method essentially does very simplistic clustering and does it fast by bypassing any nearest neighbor searches. A better result may be achieved using either an EM based fast clustering routine or by finding regions in the depth image and removing them if they don't meet some size criteria. I would probably do the latter if I had more time.

[aravindpreshant]

Thanks a lot for the input @safijari. Yes we are using ROS. It would be great if you could share the code snippet.

[chwimmer]

I'm facing the same problem currently, so I would also be interested in the code snippet. For me the highest computationally intensive part was the conversion from a PointCloud2 to a pcl::PointCloud - Type. So I wrote my own conversion and its working considerably better, but because of using three cameras and merging the pointclouds its still not good enough. But perhaps its also for you a hint how you can improve your performance.

[safijari]

Hi @aravindpreshant and @chwimmer. Sorry about the delay. The code for median filtering is here and the code for downsampling/denoising using the voxel filter is here. I hope this helps in some way.

[safijari]

@chwimmer would you consider sharing your conversion? Perhaps posting about it in the PCL mailing list :) ?

[leslysandra]

hi @mirzashah I am trying to improve the image quality of DEPTH for R200 in Linux with Librealsense. Any clue? thanks

[mirzashah]

@l3s777 that's kind of vague -- quality of what? We were able to filter out the noise as described above. If you're using librealsense directly and not the ROS driver, you should post your question at github.com/intelrealsense/librealsense

[reaganlo]

@mirzashah I see that you came up with a solution for this issue. Can this be closed?

[mirzashah]

@reaganlo We have a solution that works for our use case, but is not ideal as there is a lot of information loss due to downsampling. As stated before, there are many use cases where having a noise free point cloud is important, in particular 3d navigation + motion planning.

It would be a lot more helpful for the ROS community if it works out-of-the-box -- so I don't feel it should be closed -- unless #106 fixes it. We haven't had the chance to try the new feature yet but will soon.

If #106 doesn't fix it -- my recommendation is that you guys add as part of the launch file an optional nodelet that can apply a median filter to the depth image.

[mirzashah]

@reaganlo @I0x0I -- #106 fixes the problem! Thank you guys so much :) Closing this issue