Faulty depth measurements near object edges

AlexReimann commented 8 years ago

When using the driver there is a huge noise in the point cloud at edges of objects.

Pointing at a flat surface: orbbec noise

A video of it can be found here: https://www.youtube.com/watch?v=p2L59TLhsg4

I guess some interpolation calculation is going wrong.

This was not the case with an older driver I got from the Orbbec website (the link is down now).

procopiostein commented 8 years ago

I can confirm this issue affects both ASTRA S and ASTRA cameras.

ob-tim-liu commented 8 years ago

This was not the case with an older driver I got from the Orbbec website (the link is down now).

Could you provide the link of older driver? we will check it .

ruvi-d commented 8 years ago

I too can confirm that there is a significant difference between the older driver that was hosted on dropbox and this one. I am attaching the old driver for your reference: OpenNI-2-Linux-x64-Orbbec3D.tar.gz

tfoote commented 8 years ago

These artifacts are a challenge for navigating with the TurtleBot. It generally works but with noteably lower performance due to the extra noise needing to be rejected. And the avoidance of phantom obstacles.

Here's a screenshot from my debugging.

astra_ray_noise

It would be really great to add the filters to the released version of the driver.

bit-pirate commented 8 years ago

A bugfix for this is required before we can consider integrating the Orbbec Astra with our products. Any idea when we could expect a fix for this?

JoshBlake commented 8 years ago

@bit-pirate The binary releases of our openni2 drivers include this filter. You can find them as part of Astra SDK's openni2 folder or at https://3dclub.orbbec3d.com/t/full-openni-2-environment-download/34. Note that ARM builds exclude the filter due to the limited processing power of most ARM CPUs. If you need additional support on this please contact david@orbbec3d.com.

bit-pirate commented 8 years ago

@tfoote is the OpenNI2 bundle linked above by @JoshBlake included in our ROS package for the Astra? "Our ROS package" being your fork, if I understand the current situation correctly.

tfoote commented 8 years ago

All my changes are merged upstream. https://github.com/tfoote/ros_astra_camera/network I did the release from my branch for speed only. All development and the next release should come from the orbbec main branch.

AlexReimann commented 8 years ago

Looks like it the filter is currently in the filterlibrary branch and I guess will be eventually merged Looks much better with that.

Any chance to get any info on how this filter works?

ob-tim-liu commented 8 years ago

@AlexReimann the branch of filerlibrary is a temporary solution , that provider function same to master with extra filter driver . but it will not merged to the master. you can use codes by reference to README.

AlexReimann commented 8 years ago

So what's the planned solution?

mintar commented 7 years ago

Yup, waiting for a solution as well... any news?

AlexReimann commented 7 years ago

Bump ._.'

bit-pirate commented 7 years ago

@JoshBlake @ob-tim-liu it would be very important for us and our products, like the TurtleBot, to get a fix for this edge noise into the released driver. Could you let us know about the progress?

ob-tim-liu commented 7 years ago

now, we provider the pull request to tfoot about orbbec driver with soft filter library. before merge this request , you can use the 'filterlibrary' branch , the only different is astra_openni2 in the local dir.

刘华座机：0755-86329228 邮箱：liuhua@orbbec.com 手机：13603061506 深圳奥比中光科技有限公司 www.orbbec.com.cn 地址：深圳市南山区粤兴三道8号中国地质大学产学研基地中地大楼A808

From: Marcus Liebhardt Date: 2016-11-04 14:42 To: orbbec/ros_astra_camera CC: ob-tim-liu; Mention Subject: Re: [orbbec/ros_astra_camera] Faulty depth measurements near object edges (#7) @JoshBlake @ob-tim-liu it would be very important for us and our products, like the TurtleBot, to get a fix for this edge noise into the released driver. Could you let us know about the progress? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

mintar commented 7 years ago

In case people are wondering, @ob-tim-liu is talking about tfoote/OpenNI2#1 . Once that is merged, the master branch in this repo should do the same as the filterlibrary branch. The only difference is that the filterlibrary branch contains a copy of tfoote/OpenNI2, whereas master uses tfoote/OpenNI2 as an external dependency (which is better, of course).

bit-pirate commented 7 years ago

Understood. Thanks for the update, guys.

@tfoote could you notify us here when that PR has been merged and the package re-released? Then we will test it again (and hopefully close the issue).

AlexReimann commented 7 years ago

@tfoote bounced this back

So.. bump :p

bit-pirate commented 7 years ago

@ob-tim-liu as @tfoote mentioned in his comment in the open PR (tfoote/OpenNI2#1), Orbbec should start release their ROS S/W drivers from their own, official repo.

Could you please do the required changes and release the important changes in the filterlibrary branch?

ob-tim-liu commented 7 years ago

@bit-pirate i have change the link of OpenNI2 from tfoote's github to orbbec, you can use the latest release on master branch

bit-pirate commented 7 years ago

Thanks @ob-tim-liu ! @AlexReimann please verify.

AlexReimann commented 7 years ago

Latest deb still doesn't have the filter (not released yet?) and the master branch here doesn't show any depth image since the last commit with our Astras.

If I roll-back the master branch one commit, the filter is in and works.

So just need to release the deb and fix that issue?

bit-pirate commented 7 years ago

The release deb is still at version 0.1.5. So the commits from Sep 22 to Dec 16 are not released yet.

I'd say, 1. fix depth image issue and 2. release the softfilter changes and depth image bugfix with version 0.1.6.

mintar commented 7 years ago

Also, the deb is still released from tfoote/ros_astra_camera, not from this repo. Given the low responsiveness of the Orbbec people, I think I would even prefer it if somebody other than them would maintain the deb releases. We would just need somebody to volunteer for the job.

mintar commented 7 years ago

Also, the filter library is not really a fix, but a horrible workaround. It's so computationally expensive that it's impossible to do on an ARM board, and since it's baked into the OpenNI driver instead of a standalone node you can't run it on an external PC either.

This is what I have observed so far:

The random pixels in the depth image only occur near object edges, and only if depth registration is enabled in the driver.
They seem truly random, not interpolated between foreground and background.

Depth registration has to reproject each depth value from the depth into the rgb frame. My theory is that the random pixels occur in all pixels that are visible from the rgb frame, but not the depth frame (hence near object edges). Probably they just allocated some uninitialized memory for the registered depth image, then overwrote each pixel for which they can compute a depth value, leaving the rest uninitialized (i.e. "random").

The proper (and easy!) fix would be to just initialize the registered depth image with "NaN". If Orbbec could upgrade their firmware to do just that, we wouldn't need the filter library any more, and the Astra would finally become usable.

wjwwood commented 7 years ago

I don't know how I can help, but I wanted to add another data point (and to make sure what I'm seeing isn't a new issue) I get something like this when the astra is too close to an object or it is looking at a surface that is out of range:

screenshot from 2017-02-24 15-37-52

Here's a gif of me moving it and show it in a glass door and if I put my hand close: https://gfycat.com/MelodicBlondKudu

Notice the left-right smears in the depth image.

ob-tim-liu commented 7 years ago

you should use the filter-library to solve this problem

刘华座机：0755-86329228 邮箱：liuhua@orbbec.com 手机：13603061506 深圳奥比中光科技有限公司 www.orbbec.com.cn 地址：深圳市南山区粤兴三道8号中国地质大学产学研基地中地大楼A808

From: William Woodall Date: 2017-02-25 07:47 To: orbbec/ros_astra_camera CC: ob-tim-liu; Mention Subject: Re: [orbbec/ros_astra_camera] Faulty depth measurements near object edges (#7) I don't know how I can help, but I wanted to add another data point (and to make sure what I'm seeing isn't a new issue) I get something like this when the astra is too close to an object or it is looking at a surface that is out of range: Here's a gif of me moving it and show it in a glass door and if I put my hand close: https://gfycat.com/MelodicBlondKudu Notice the left-right smears in the depth image. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

tfoote commented 7 years ago

@ob-tim-liu Most users use the prebuilt debian packages in which the filter library is not included. A new release which includes the filter here: https://github.com/tfoote/OpenNI2/pull/1/files would give most users a much better experience with the astra when used with ROS.

AlexReimann commented 7 years ago

@mintar we get the problem also when there is no depth registration

Update: The problem was one of our engineers was using the wrong repository / branch

Besides that we run into a new problem case:

sensor_noise1

sensor_noise2

In this case we get noise depth pixels up to a certain pixel height (you can clearly see that in the lower right corner) in the areas where the sensor can't get a proper depth value. This looks to me like some memory initialization or corruption problem.

I agree on @mintar that fixing this by fixing the firmware or driver properly is the way to go instead of hiding the problem by using some filter.

@ob-tim-liu Please could you let someone make sure you are properly initializing stuff (I'm humbly suggesting memcheck)?

I will do some more test over the day and see if there is a way around this.

Update: The problem was one of our engineers was using the wrong repository / branch

alberto-tellaeche commented 7 years ago

Hello all; After having problems with the astra driver from indigo repositories and having issues, I have compiled this driver from source testing the flag -DFILTER to both values ON and OFF on filter branch of this repository. Seems to work better. In any case, any plans to move these changes to master? Is an alternative fix planned for the package in ROS or an alternative solution in Astra FW?

Thanks, Alberto

aswinsarang commented 7 years ago

astra 1 Is there a bug fix?

ob-jackson commented 7 years ago

@aswinsarang Please use the filter library from the above comments.

AlexReimann commented 7 years ago

The problem stated 4 posts above was because someone used a branch / repository without the filter library.

facontidavide commented 4 years ago

I wonder if anyone confirmed that the problem was actually a not-initialized buffer or not.

And about the filter...where can I find the source code or the algorithm to see if we can do any better?

orbbec / ros_astra_camera

Faulty depth measurements near object edges #7