How to extend AR Foundation to access Scene Depth Map

[sam598]

So instead of demanding that Unity immediately implement the new features for ARKit 4 that were released during WWDC today (as is tradition), I decided to try and see how far I can get with Extending AR Foundation through the session's native pointer: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/extensions.html

The specific feature I am trying to access is the new "scene depth" texture using the ARKit Lidar camera. https://developer.apple.com/documentation/arkit/creating_a_fog_effect_using_scene_depth

I am sure that there are updates coming as soon as possible, so this is more of an exercise than a long term solution. But I thought this question might help others try to figure out how to extend AR Foundation.

This is my code so far:

ARDepthMap.mm

#import <ARKit/ARKit.h>

typedef struct UnityXRNativeSessionPtr
{
    int version;
    void* session;
} UnityXRNativeSessionPtr;

extern "C"
{
    void UnityEnableSceneDepth(UnityXRNativeSessionPtr* nativeSession)
    {
        ARSession* session = (__bridge ARSession*)nativeSession->session;

        ARConfiguration* config = session.configuration;

        config.frameSemantics = ARFrameSemanticSceneDepth;

        [session runWithConfiguration:config ];
    }

    CVPixelBufferRef UnityGetDepthMap(UnityXRNativeSessionPtr* nativeSession)
    {
        ARSession* session = (__bridge ARSession*)nativeSession->session;
        return session.currentFrame.sceneDepth.depthMap;
    }
}

ARDepthMap.cs

using System;
using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class ARDepthMap : MonoBehaviour
{
    [DllImport("__Internal")]
    public static extern void UnityEnableSceneDepth(IntPtr ptr);

    [DllImport("__Internal")]
    public static extern XRCpuImage UnityGetDepthMap(IntPtr ptr);

    public ARSession arSession;

    public void EnableSceneDepth()
    {
        UnityEnableSceneDepth(arSession.subsystem.nativePtr);
    }

    public void GetDepthMap()
    {
        XRCpuImage depthmap = UnityGetDepthMap(arSession.subsystem.nativePtr);
    }
}

The first step is to modify the session configuration to support semantic scene depth. This seems to work fine even after a session has already been started.

The second step is to get the depth map from the current frame's sceneDepth object. Based off of this issue https://github.com/Unity-Technologies/arfoundation-samples/issues/347 I naively tried to cast CVPixelBufferRef as XRCpuImage. But that (understandably) did not work.

I thought about trying to see how XRCameraSubsystem.TryGetLatestFrame works, however that seems to be completely closed source as well. As is XRCpuImage.Transformation.Convert.

I could try to create a metal texture in objective c and pass it to Unity. However I would like to try to keep this consistent with how AR Foundation is handling the rest of the textures in the latest versions.

Is there any available documentation or code to convert CVPixelBufferRef to XRCpuImage? Or any suggestions on how to efficiently handle a CVPixelBuffer on the C# side of Unity?

[tdmowrer]

Support for environment depth has been published as a preview package (4.1.0-preview.1). This includes XRCpuImage support.

[tdmowrer]

We just published a blog post announcing support for ARKit 4.

[sam598]

But it took the AR Foundation team almost 48 hours to implement these new features? Unacceptable!

Thanks @tdmowrer, really appreciate it! Very excited to work with these features.