MediaFrame, Camera capture

[cwule]

I could not find any info on that, but is there any functionality with OpenXR to grab mediaframes together with their respective SpatialCoordinateSystem? This is necessary for use with the locatable camera. I especially need the transformations from camera to world space.

[yl-msft]

@cwule

You can use the OpenXR XR_MSFT_spatial_graph_bridge extension to locate the camera on HoloLens 2.

We've not publish the sample code on this yet, but here are some code snippet for you to get started. We will clean up our internal sample and publish here to github soon.

Here's how to find the dynamic node Id from the MediaFoundation winrt APIs MediaFrameReference.

    std::optional<std::pair<winrt::guid, XrPosef>> FindDynamicNode(const MediaFrameReference& frame) {
        // MFStreamExtension_CameraExtrinsics
        constexpr winrt::guid CameraExtrinsicsGuid(0x6b761658, 0xb7ec, 0x4c3b, {0x82, 0x25, 0x86, 0x23, 0xca, 0xbe, 0xc3, 0x1d});
        if (auto inspectable = frame.Properties().TryLookup(CameraExtrinsicsGuid)) {
            auto propertyValue = inspectable.try_as<winrt::Windows::Foundation::IPropertyValue>();
            if (propertyValue && propertyValue.Type() == winrt::Windows::Foundation::PropertyType::UInt8Array) {
                winrt::com_array<uint8_t> bytes;
                propertyValue.GetUInt8Array(bytes);
                if (bytes.size() >= sizeof(MFCameraExtrinsics)) {
                    const auto* const extrinsics = reinterpret_cast<const MFCameraExtrinsics*>(bytes.data());
                    if (extrinsics->TransformCount > 0) {
                        const MFCameraExtrinsic_CalibratedTransform& transform = extrinsics->CalibratedTransforms[0];
                        winrt::guid dynamicNodeId;
                        xr::CopyGuid(dynamicNodeId, transform.CalibrationId);
                        return std::make_pair(dynamicNodeId, xr::math::Pose::MakePose(transform.Orientation, transform.Position));
                    }
                }
            }
        }
        return std::nullopt;
    }

Here's how to create an OpenXR space handle from the dynamic node id.

        void CreateCameraXrSpace(const std::pair<winrt::guid, XrPosef>& dynamicNode) {
            const auto& [nodeId, pose] = dynamicNode;
            XrSpatialGraphNodeSpaceCreateInfoMSFT spaceCreateInfo{XR_TYPE_SPATIAL_GRAPH_NODE_SPACE_CREATE_INFO_MSFT};
            spaceCreateInfo.nodeType = XR_SPATIAL_GRAPH_NODE_TYPE_DYNAMIC_MSFT;
            xr::CopyGuid(spaceCreateInfo.nodeId, nodeId);
            spaceCreateInfo.pose = pose;
            CHECK_XRCMD(
                m_context.Extensions.xrCreateSpatialGraphNodeSpaceMSFT(m_context.Session.Handle, &spaceCreateInfo, m_cameraSpace.Put()));
        }

[cwule]

Thanks a lot, I will look into that!