search

googlesamples / arcore-depth-lab

ARCore Depth Lab is a set of Depth API samples that provides assets using depth for advanced geometry-aware features in AR interaction and rendering. (UIST 2020)
https://augmentedperception.github.io/depthlab/
Apache License 2.0
762 stars 152 forks source link

Depth-lab in WebXR #42

Closed akssieg closed 12 months ago

akssieg commented 2 years ago

Can we use arcore-depth-lab with WebXR? I want to try 3d cursor which uses arcore-depth-lab for accurate placement and orientation of the raticle.

ruofeidu commented 2 years ago

Hi Akssieg,

I just learnt that it is possible right now!!

If you try out the "AR Depth Sensing" demos on this page they seem to work correctly: https://storage.googleapis.com/chromium-webxr-test/r991081/proposals/index.html

Then you may combine this with three.js and translate our C# code into JavaScript, maybe it's high time to make a JS library for DepthLab and I always welcome PRs!

Cheers, Ruofei

akssieg commented 2 years ago

Hi Ruofeidu,

Thanks for providing the pointer to the WebXR proposals.

Regards, AK

akssieg commented 2 years ago

@ruofeidu I am following your suggestion on code translation from C# to JavaScript but how to estimate DepthSource.FocalLength.x ? I don't see any intrinsic matrix in the data structure of https://www.w3.org/TR/webxr-depth-sensing-1/#xr-cpu-depth-info-section for WebXR.

akssieg commented 2 years ago

Is Depthinfo.normDepthBufferFromNormView a projection matrix? Can I use it to estimate focal length?

ruofeidu commented 2 years ago

Great discussion here!

I honestly have little experience with WebXR depth but three.js.

Intrinsic matrix is not that important, you may run DepthLab to obtain it and copy focal length etc. to a web demo - it should roughly align with other mobile phones... I don't see intrinsics being exposed by quickly reading https://www.w3.org/TR/webxr-ar-module-1/

Also check out my recent 3D photo demo in TF.js using the DepthLab 3D photo algorithm: https://storage.googleapis.com/tfjs-models/demos/3dphoto/index.html

The intrinsics are heuristic but work well.