search

googlesamples / arcore-depth-lab

ARCore Depth Lab is a set of Depth API samples that provides assets using depth for advanced geometry-aware features in AR interaction and rendering. (UIST 2020)
https://augmentedperception.github.io/depthlab/
Apache License 2.0
762 stars 152 forks source link

Continuous evironment mesh building #31

Closed bitk0der closed 12 months ago

bitk0der commented 3 years ago

Hello, first of all - thank you for such a great project!

I have a question about continuous evironment mesh building - could you point me a way how to implement creating enviroment "scanner", i want to create approximate model of around environment - i see this like every new frame will fill and deform already existing "scanned" mesh and make it more accurate (in future - with environment texture mapping). I think the simpliest way is to subtract new scanned mesh from already scanned.

Maybe it can be a good and very interesting objective for a new sample!)

ruofeidu commented 3 years ago

Yes, I agree that an interesting objective is to reimplement this! My pointers are the KinectFusion papers and follow up works:

  1. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/kinectfusion-uist-comp.pdf
  2. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ismar2011.pdf
  3. https://www.youtube.com/watch?v=KOUSSlKUJ-A

The current version of DepthLab does not support continuous reconstruction yet, but there is potential. You can also see this https://youtu.be/EerbBxLt7hI?t=48 that leveraged our raw Depth API as a partner.

Ademord commented 3 years ago

hello @bitk0der i am doing something similar to what you describe, could we talk on discord and maybe help each other? ademord#2999

agent00jackson commented 2 years ago

@Ademord Sent you a friend request on discord, I am looking at doing something similar. I have found info on using Open3d libraries to construct meshes, but I lack the knowledge to integrate it with Unity. I don't necessarily need mesh generation to be continuous and real time, but I would need to still generate meshes "in-game". My thought process was that I could save the point cloud, generate a mesh, and allow the user to manipulate the mesh.

ruofeidu commented 12 months ago

I'll close it for now until I have further update in continuous mesh, btw, if you are interested in 3D photos with depth mesh, check out the demo in https://visualblocks.withgoogle.com/#/edit/_?project_json=https:%2F%2Fraw.githubusercontent.com%2Fgoogle%2Fvisualblocks%2Fmain%2Fpipelines%2Fgraphics%2F3dphoto_portrait_depth.json

more demos in https://github.com/google/visualblocks/tree/main/pipelines