Non-rigid Reconstruction pipeline

[fzy139]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

Proposed new feature or change

Non-rigid reconstruction aims to model the dynamic scenes or objects while the traditional 3D reconstruction can be only used for static scenes. It takes RGBD sequences as input and outputs a canonical model with a warp field that encodes deformation over time. Volumetric video, holoportation and many other amazing techniques are based on non-rigid reconstruction. Amounts of methods have been developed in recent years since the seminal DynamicFusion (extended from KinectFusion), including learning-based methods.

In Open3D, there are already some features that are key components of common non-rigid reconstruction pipeline, such as TSDFVolume or ARAP deformation method. It would be much feasible to integrate a whole non-rigid reconstruction with some new features such as:

1. deformation graph or warp field
2. non-rigid ICP

And currently most open-sourced non-rigid reconstruction implementation are broken due to old cuda driver or optimization solver. So Open3D will make a big difference in this field if new features and pipelines were developed.

References

1. Newcombe et al. CVPR'2015 DynamicFusion paper
2. SurfelWarp
3. DeepDeform CVPR'2020

Additional information

surfelwarp repo is an available open-sourced implementation now.

[theNded]

Thanks for the advice. Unfortunately we do not have the bandwidth to develop such a challenging feature. As of now we have SLAC implemented, which does volumetric deformation for static scene undistortion. An extension after experiments might be possible. You may refer to the VoxelBlockGrid and ControlGrid for implementation details. We welcome contributions.