fabiencastan
commented
3 years ago Hi @SamMaoYS ,
Very cool work! It would be good to setup a confcall to discuss about it in an interactive session. Can you contact me at fabien.castan@mikrosimage.com?
If the depth maps are good but the fusion is not, it may be a problem with the accuracy of the cameras estimation. You can try to make the SfM in Meshroom from the source images, then use the SfMAlignment node to align this result to the sfmData you have created from the iPad information. And then launch the Meshing on that using the precomputed depth maps. So you will be in the right scale to use the depthmaps from the sensor, but the cameras alignment will be based on the source images (and not from the realtime image+imu fusion).
You can also try to see the impact if you initialize the SfM node from your initial sfmData (in all case you need the SfMAlignment node to realign at the end).
SamMaoYS
remmel
My next steps will be to fine tune the scale, test differents intrinsics and to improve calculated depthmaps instead of remplacing it by tof depthmaps
to calculated the ratio size between meshroom and world, I make sure that 1st picture has in its center an image which has both features and 100% Tof confidence, and later compare depth or centered pixel (w/2,h/2). I do not try to import anymore AREngine poses, as thoses poses are not perfect neither (in a test, the mesh generated importing poses with worst than using the default pipeline)
Meshing.estimateSpaceFromSfm must be uncheck to really see the difference, otherwise the bounding box is too small
MAdelElbayumi
stale[bot]
VihangMhatre
natowi
Describe the problem Hi, thanks to the wiki and previously solved issues, I am able to reconstruct scenes with known camera poses. Since it is hard to predict the depth values in the untextured areas, I want to further improve the result by utilizing depth maps from a depth sensor. I manage to do it by replacing the depth values of the filtered depth maps in node DepthMapsFiltering node with the depth values from the sensor. Since I also reconstruct with known camera poses, the depth values should be correct. Overall, this process works as expected, but the reconstructed mesh has some significant artifacts. Holes, bumpy surfaces. What I know is during the meshing, Delaunay tetrahedralization is first used to do the volume discretization, then weights are assigned to the vertices, finally a minimal s-t graph cut to select the surfaces. Screenshots Projected dense point cloud from all depth maps (depth value from the sensor).
Meshing result from the above point cloud.
Projected dense point cloud from all depth maps (depth value estimated by meshroom pipeline)
Meshing result from the above point cloud.
Dataset The images (RGB + depth + camera poses) are collected from an iPad, using Apple ARKit framework. The images are decoded from a video stream, so they are blurry. The meshing process, however, does not really depend on triangular feature points, so this should not be a big problem.
Desktop (please complete the following and other pertinent information):
Question My question is what could be the reason that with depth values from the depth sensor, the reconstructed meshes have many holes. One thing to add is that the depth maps from the sensor is not very accurate. But is much better comparing to the depths estimated by meshroom in untextured areas. I can tweak the parameters of Nb Pixel Size Behind, Full Weight to fill those holes, but result in less details in the geometry.
Depth comparison (Left, meshroom estimated depth; Right, depth from sensor) Textured object
Limited Textures
Bumpy surfaces when I try to reconstruct with sensor depths (reconstruction of a laundry room)
Is it because of the projected point cloud is too dense? I notice the point cloud is augmented with silhouette points. If the point cloud is too dense, will it influence the accuracy of the s-t graph cut? Overall, I am trying to find the reason why the reconstructed meshes is less complete, less smooth on the surfaces. If you need more information, please let me know. I will try to clarify as many questions as possible. Thank you!