search

AljazBozic / DeepDeform

[CVPR'2020] DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data
183 stars 17 forks source link

Generating mask for every frame #5

Closed zhan-xu closed 3 years ago

zhan-xu commented 3 years ago

Hello! Thanks for the dataset and toolbox! I am wondering is there a way to generate object mask for every single frame, so that I can have a continuous scanned sequence of a particular object? Thanks!

shubhMaheshwari commented 3 years ago

@AljazBozic can you provide some insights and algorithms that could be used for generating mask for every frame?
Specifically what algorithm was used to generate masks in the paper?

AljazBozic commented 3 years ago

Hi @zhan-xu and @shubhMaheshwari! Thank you for your interest! Actually, at the test-time non-rigid reconstruction the object masks are not used, only the mask of first frame is assumed (and given, it was manually annotated), to know which object to reconstruct. Because the reconstruction method reconstructs the object incrementally, the background doesn't get integrated, unless there is some strong and persistent contact with the background (it relates to how deformation graph is expanded in a stable way). So the masks were not used in the reconstruction algorithm. I'll try to take some time to upload the non-rigid reconstruction framework, then it would be easier to know the details.

shubhMaheshwari commented 3 years ago

Just want to clarify. Using the initial mask, the deformable graph(embedded graph) is constructed. Now during TSDF integration, the TSDF value and weights are updated only for voxels which are affected(determined by pixel anchors) by the nodes of deformable graph are updated. After integration, marching cubes outputs the canonical model using which the deformable graph is updated. In the original dynamic fusion paper, vertices of the canonical model which are outside the node coverage are subsampled and these subsampled vertices are newly added nodes to the deformable graph.

Is this correct ?

AljazBozic commented 3 years ago

Yes, exactly, the integration is selective, so only around existing nodes (since you need to have an estimate of deformation from canonical space to current frame, and you estimate the deformation only for existing nodes). But after integration, the region around nodes grows, and after extracting updated mesh with Marching cubes, some vertices might not be covered by existing nodes. Then new nodes are computed by sub-sampling non-covered vertices.