search

googlesamples / arcore-depth-lab

ARCore Depth Lab is a set of Depth API samples that provides assets using depth for advanced geometry-aware features in AR interaction and rendering. (UIST 2020)
https://augmentedperception.github.io/depthlab/
Apache License 2.0
780 stars 153 forks source link

How does Depth Lab provide 0-65m depth values? #47

Closed valantano closed 1 year ago

valantano commented 1 year ago

I noticed that the Depth Lab uses AR Foundation v.4 but provides depth estimation for 0-65m, although AR Foundation v4 does not support this. Currently, I am working with AR Foundation v.5 since I need to work with the whole 0-65m depth estimation, but I also need to work with the ARCore Extensions, which are only available for AR Foundation v.4. When I try to use AR Foundation v.4, I cannot access the 0-65m depth estimation but only the 0-8m depth estimation since everything above 8m is set to 0. However, when using the Depth Lab app and activating the Depth Map, it seems that Depth Lab is able to detect the depth even for higher values than 8m. I looked into https://github.com/googlesamples/arcore-depth-lab/blob/22cd7f1ce4eb2ed73bda19a0ea1bf3e636831799/Assets/ARRealismDemos/Common/Scripts/DepthSource.cs, but I don't get it why Depth Lab is able to provide the 65m depth values.

It would be great if someone could help me with that. Maybe someone can explain where the values greater than 8m are set to 0 and how Depth Lab bypasses this. And perhaps also how I can do this as well.

I really appreciate any help you can provide.

ruofeidu commented 1 year ago

Hi valantano,

I updated DepthLab legacy SDK (deprecated) to 30m depth sensing around Google I/O. We have an engineer updating DepthLab AR Foundation SDK but it's not complete yet. The paper and the Play Store app were built with the legacy SDK. If you can get normalized depth from 0 to 1 without the 8 meter clipping, would you like to try just converting it to 0-30m?

I have an updated shader here:

https://github.com/googlesamples/arcore-depth-lab/blob/arcore_unity_sdk/Assets/ARRealismDemos/DepthEffects/Shaders/BackgroundToDepthMapCore.cginc#L48

and a new fine-tuned colormap :) : https://github.com/googlesamples/arcore-depth-lab/blob/arcore_unity_sdk/Assets/ARRealismDemos/Common/Textures/DepthColorRamp.png

Thanks, Ruofei

valantano commented 1 year ago

Hi Ruofei,

thank you for the fast response. Sadly, as soon as I know, AR Foundation only provides the Depth in meters in a texture. And there they have already set all values above 8m to 0. At least in AR Foundation v.4. It seems that I will need to use AR Foundation v.5. But your explanation helped me to understand the reason why you guys are able to provide the 30m in Depth Lab :)

Thanks, Valentino

ruofeidu commented 1 year ago

Let me reopen this and ask relevant engineers tomorrow! Thanks for letting me know!

ruofeidu commented 1 year ago

Hi Valentino,

Please check out https://docs.unity.cn/Packages/com.unity.xr.arcore@4.1/changelog/CHANGELOG.html and from 4.1.0-preview.13, it switches to using all 16-bits from the pixels in the ARCore environment depth image, rather just the lower 13 bits :)

Cheers, Ruofei