search

Unity-Technologies / arfoundation-samples

Example content for Unity projects based on AR Foundation
Other
3.07k stars 1.15k forks source link

Modifications to native depth map images are undocumented #616

Closed sam598 closed 3 years ago

sam598 commented 4 years ago

According to this comment https://github.com/Unity-Technologies/arfoundation-samples/issues/190#issuecomment-693542595 there may be undocumented changes being done to depth maps coming from native device APIs:

we do a fair amount of smoothing on the depth image already

It is not entirely clear what this comment means. If Unity is doing additional post processing on depth maps that can lead to an undesirable and unrecoverable loss in depth map quality.

I would like to see better documentation about which depth map settings correspond to unmodified images from the native API (e.g. sceneDepth and smoothedSceneDepth), which (if any) are being post-processed by Unity, and how they are being modified.

tdmowrer commented 4 years ago

The exact post-processing is subject to change and varies by platform, but one thing we could clarify in our docs is that the "fastest" mode always mean "no processing". So that will give you the unmodified image.

sceneDepth and smoothedSceneDepth

Note that the smoothing we do is a per-frame smoothing of the sceneDepth while ARKit's smoothedSceneDepth is a temporal smoothing. It is intended for static objects; if an object were to move, you would get a sort of "after image" effect.

sam598 commented 4 years ago

one thing we could clarify in our docs is that the "fastest" mode always mean "no processing". So that will give you the unmodified image.

That would be great.

Note that the smooth we do is a per-frame smoothing of the sceneDepth while ARKit's smoothedSceneDepth is a temporal smoothing.

Understood. I would still recommend making different options from the native device accessible and clearly labelled, so that the consumer of the AR Foundation API can choose what is best for their application or use case (e.g. fast occlusion vs object scanning).

rdtredinnick commented 3 years ago

Following up on this - I am wondering if there are any plans to expose "smoothedSceneDepth" values directly through ARFoundation? Perhaps a flag on the session or something that would then mean TryAcquireEnvironmentDepthCpuImage and TryAcquireEnvironmentDepthConfidenceCpuImage returns smoothedSceneDepth values instead of sceneDepth.

tdmowrer commented 3 years ago

4.2.0-pre.8 now supports the "smoothedSceneDepth" option (we call it "temporal smoothing" in the AROcclusionManager).