How can I get world space ray from pv frame ?

[kaiwu119]

I want get a ray from pv frame pixel coordinats, first I use as follow to get a ray in pv camera space : pvFrame->SensorStreamingCameraIntrinsics->MapImagePointToCameraUnitPlane(uv, &xy); but it not work, when I deploy app to hololen, the app crashes directly(I only add one line, as bold code)

    DepthPvMapper::DepthPvMapper(HoloLensForCV::SensorFrame^ depthFrame, HoloLensForCV::SensorFrame^ pvFrame)
    {
        _imageToCameraMapping = createImageToCamMapping(depthFrame);
        **_pvImageToCameraMapping = createImageToCamMapping(pvFrame);**

    }

cv::Mat DepthPvMapper::createImageToCamMapping(HoloLensForCV::SensorFrame^ sensorFrame)
    {
        cv::Mat imageToCameraMapping = cv::Mat(
            sensorFrame->SoftwareBitmap->PixelHeight,
            sensorFrame->SoftwareBitmap->PixelWidth,
            CV_32FC2,
            cv::Scalar::all(0));
        for (int x = 0; x < sensorFrame->SoftwareBitmap->PixelWidth; ++x) {
            for (int y = 0; y < sensorFrame->SoftwareBitmap->PixelHeight; ++y) {
                Windows::Foundation::Point uv = { float(x), float(y) };
                Windows::Foundation::Point xy(0, 0);

                if (sensorFrame->SensorStreamingCameraIntrinsics->MapImagePointToCameraUnitPlane(uv, &xy)) {
                    imageToCameraMapping.at<cv::Vec2f>(y, x) = cv::Vec2f(xy.X, xy.Y);
                }
            }
        }
        return imageToCameraMapping;
    }

I also try to get the inverse of the projection matrix of pv frame，this will also crash the program. Can someone help me ？ Thanks in advance！

[emeraldy]

Hi,

I think the pv camera uses a different function for unprojection: there is a CoreCameraIntrinsics object in the SensorFrame class, which provides a method called UnprojectAtUnitDepth. Try this and you should get a ray in the pv camera space to start with. Since the camera is looking down at -z, the unit plane's z assumed by the function is -1. The documentation is here: https://docs.microsoft.com/en-us/uwp/api/windows.media.devices.core.cameraintrinsics.unprojectatunitdepth?view=winrt-19041#Windows_Media_Devices_Core_CameraIntrinsics_UnprojectAtUnitDepth_Windows_Foundation_Point_

[kaiwu119]

Thank you very much for your answer, this gave me a direction to try, thanks again.