How to display yolo labels on both eyes in a VR Headset ?

[Blindsp0t-creative]

Hi,

I have Yolo working fine with my Zed mini, and it's awesome.

I'm wondering how to display the labels on detected objects in both left and right eyes in a VR Headset without analyzing the video feeds of each camera ?

Is there a way to compute a correspondance for a given pixel in the two cameras ... ?

Any help will be appreciated !

[adujardin]

Hi, Yes, there's a way since the cameras are calibrated. Each line of the left image corresponds to the line in the right image. We can use this: we will find the correspondence of the object centroid into the right image and apply the difference to the 4 corners.

Let's consider the 3D centroid of an object in the left image (X_left, Y_left, Z_left) and the 2D centroid (U_left, V_left)

1. Compute the centroid disparity : disparity_centroid_left = (baseline * focal_x) / Z_left

2. 2D right centroid : U_right = U_left - disparity_centroid_left V_right = V_left

We can apply the same offset to each corner (or compute it with the 2D centroid and box dimension)

The baseline and focal can be obtained with (API Doc link): focal_x = Camera::getCameraInformation().calibration_parameters.left_cam.fx baseline = Camera::getCameraInformation().calibration_parameters.T.x

[Blindsp0t-creative]

Super nice, thank you for your answer.

I'm not sure how to compute the Z_left, do you compute it from the depth map...?

[adujardin]

Using the ZED with Yolo will output both 2D boxes and 3D centroid (here)

The extraction function is here : https://github.com/AlexeyAB/darknet/blob/42d08fd820335584365d393da3967853676a8c35/src/yolo_console_dll.cpp#L38-L93

Z_left is contained in bbox_t::z_3d

[Blindsp0t-creative]

Thank you again for the clarity and responsiveness.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days