Passthrough Camera Projection?

[ddiakopoulos]

Are there any tricks to applying the projection or pose from tracked camera views?

trackedCamera->GetCameraProjection(vr::k_unTrackedDeviceIndex_Hmd, vr::VRTrackedCameraFrameType_MaximumUndistorted, near, far, &trackedCameraProjection);

Assuming that the chaperone tron overlay is a mathematically ideal way of projecting the pass through camera image -- I've gotten close. Rendered here is my application's version of the camera view with Steam's chaperone overlayed (screenshot taken from mirror window). But it's not quite what the user sees with just the overlay.

[natbro]

sorry for the slow reply. I may not grok your entire case, but you may need to adjust the camera projection based on the camera extrinsic, vr::Prop_CameraToHeadTransform_Matrix34 from the device.

[epajarre]

Joining this slow conversation (sorry)

I don't really have the correct answer, but maybe if I write my opinion somebody will correct me. In any case googling related to this issue, has given me very little information.

I think that instead of using the Prop_CameraToHeadTransform_Matrix34 you can also just use the standingTrackedDevicePose.mDeviceToAbsoluteTracking field from the "camera_header" which you can get using the GetVideoStreamFrameBuffer call

The other thing I recently learned that I can use the inverse of the matrix from the GetCameraProjection to scale, and final positioning of the textured quad with the camera image. Just have to remember that the projection is apparently an D3D projection so (as an OpenGL user) I just have to draw my rectangle from (-1,-1,0) to (1,1,0) after this setup.

Now the resulting image does not quite match the Steam chaperone display, but as far as I have been able to verify by staring my hand and screen through the HUD and the removing the helmet and trying it remember how it was... this may be caused by the fact that the Steam display is not perfect!

The last thing I learned, was that when I was using GetVideoStreamTextureGL for an "automatic" texture, and GetVideoStreamFramebuffer for just polling the camera orientation. there was a nasty lag which cased the projected view to jump behind when rotating my head. This same thing happened if I used the basic tracking information from the helmet with then Prop_CameraToHeadTranform. I am now happy when I noticed that this effect disappears, if I use the GetVideoStreamFramebuffer for both the camera orientation and for getting the raw (actually Undistorted) image at the same time, and load my texture manually.

Eero

[Frank-Dz]

Hi~ Did you successfully solve the problem?