search

isl-org / Open3D

Open3D: A Modern Library for 3D Data Processing
http://www.open3d.org
Other
11.21k stars 2.27k forks source link

Real-time mesh building #2221

Open bumbastic opened 4 years ago

bumbastic commented 4 years ago

I am trying to use Open3D for real-time generation of 3D models using a Kinect for Azure device. Now that there is CUDA support. I believe this should be possible.

To get a good start, I would appreciate any hints on how to get the most out of Open3D for this task. I must add that I already know the camera pose of each image taken. With a small tolerance of about +/-30mm, and rotation +/-2deg.

I was already hinted to drop rgbd odometry and go directly to multiway registration. This will require input images converted to point clouds + image poses, and output an optimized pose graph with optimized image poses. This can then be used as input to a TSDF that can generate the final mesh. Is this the way to go?

The Kinect for Azure SDK can generate nice stable 30fps point clouds. Could I somehow take advantage of this? The ScalableTSDFVolume Class only integrates rgbd images. Is there any way to fast register and integrate the point clouds, without needing the sensors RGBD-images for TSDF integration?

theNded commented 4 years ago

If you have groundtruth poses, you can go directly with integrate. This is non-trivial, and is a work in progress including both integration and mesh generation. Please stay tuned.

Kinect's point clouds are originally from rgbd images. It is impossible to integrate pcds correctly, see my explaination in #1877.

bumbastic commented 4 years ago

I have kind of ground truth poses. But mechanical tolerenses and parts osilating because they are hanging in wires, calls for some registration support. I've got the integration running just fine. My biggest problem right now is getting RGBD timestamps from the python api. The camera moves up to 10 meters pr. minute, making precise timestamps important.

bumbastic commented 4 years ago

About integrating pointclouds like rgdb. If the point cloud are structured with width, height and viewpoint info. from the rgbd they originated from, then it should be possible?

theNded commented 4 years ago

I think one problem is that inaccurate timestamp may also introduce error. Timestamps are not easy to synchronize, as RGB and depth are from different engines from the hardware and Kinect SDK as well, see #2203. So given the setup, the best we can do is to rerun the entire pipeline without odometry (but with pose graph between non-adjacent frames), and with pairwise registration in a sliding window (instead of exhaustive register pairs, register only 2-3 fragments away).

For the integration problem, if the point cloud has all the information, then we need to reconstruct RGB-D from point clouds first. This involves bilinear interpolation and hole-filling -- it is doable, but not pretty straight forward, see https://github.com/mp3guy/ElasticFusion

bumbastic commented 4 years ago

Thanks for the explanations. I'll start with a direct integration with ground truth poses and 'fast' moving cameras. I need to get those timestamps synced for up to four kinects, and integrate all images into one volume. I probably also need to do bilateral filtering on the depth images. When this is working I can start experimenting with some registration support.