Projection of Depth map into point cloud

[saching13]

Start with the why:

The depth map that we can currently obtain from the DepthAI API is an image that contains depth information at each index of the matrix/image.

Projecting this depth map into point cloud allows us to exploit features like point cloud's registration, subsampling, reconstruction etc.

For the initial implementation, this step is done on the host side instead of the Myriad X

It is important to do this on the Myriad X itself since it can unlock the above-mentioned features to be carried out on the Myriad X and reducing more load on the CPU.

The How:

Considering the depth map image is relative to the right camera of the stereo camera. Using its intrinsic parameters, we can obtain the (x_k, y_k, 1) in the camera reference frame. Multiplying z value from the depth map will provide us with the (x, y, z) of each point captured by the stereo camera.

The what

Support point cloud projection on the DepthAI itself, with options like

• sub-sampling
• registration

[julianaddison]

Hi @saching13,

I'm trying to extract the depth map (i.e. depth_raw stream) alongside the raw RGB data stream. I was hoping you might have some experience with extracting it given your reference to the depth map.

I've managed to save the RGB video stream via the line below and am trying to understand if both RGB and depth_raw streams can be saved similarly. Are you aware of any way I could do this?

python3 test.py -v [path/to/video.h264]

[saching13]

Currently, I have implemented it with right frame as of now. you can take a look at that here. We will soon release an update with rgb camera also. If you want to combine with rgb right away with good accuracy you need to first get the extrinsic calibration between rgb and right camera of stereo and project the depth_raw frame into the rgb camera's frame.

I am not sure about the way to save the depth_raw. I am adding @Luxonis-Brandon here for more details.

[Luxonis-Brandon]

I think @VanDavv has been working towards this using a PR.

[Luxonis-Brandon]

This is implemented in develop and an example program is here: https://github.com/luxonis/depthai-experiments/tree/master/point-cloud-projection

[Luxonis-Brandon]

Oops, missed the on the Myriad X part... reopening this issue.

[John-Dean]

Is there an estimated timeline for this?

I'm trying to turn multiple cameras into point clouds, merge them together and then perform further code on the result and I'm wondering if I should be looking to do the point cloud generation on the Myriad X (if it's at least 60fps and minimal latency) or on the host PC?

[Luxonis-Brandon]

Hi @John-Dean ,

We're making progress on the point-cloud generation on the Myriad X, but took a detour to make better ROS support, so we haven't finished this yet.

@saching13 can provide more information on this.

But anyway, I would recommend generating the point-cloud on the host for now.

Thanks, Brandon

[saching13]

Hey @John-Dean , Sorry for the delayed response. I think it is really hard to get point cloud conversion of 60fps on the device side. So Like Brandon suggested it would be better to do it on the host. And on the device will try to minimize the computation as much as possible.

I am thinking best case scenario it would take a month taking into consideration of other tasks, type handling and alignment issues. I have a CPP code here which does it host and exploits multiprocessor. This example converts it to a point cloud and then converts it back into another camera frame. you just need the first part.

[saching13]

Hey @John-Dean , Sorry for the delayed response. I think it is really hard to get point cloud conversion of 60fps on the device side. So Like Brandon suggested it would be better to do it on the host. And on the device will try to minimize the computation as much as possible.

I am thinking best case scenario it would take a month taking into consideration of other tasks, type handling and alignment issues. I have a CPP code here which does it host and exploits multiprocessor.

[saching13]

Created an host based example for rgb alignment using gen2. Code can be found here P.S: Follow the instructions in the example carefully on how to modify calibration information manually since calibration api is not available in gen2 yet.

[MOj0]

Added RGB Point Cloud Visualizer with depth information here.

[madgrizzle]

Is there any chance that device-side point cloud conversion might end up happening, even at a lower FPS and/or resolution? I'm starting to work on ROS2 navigation and think I can get by with something as low as even 15FPS + 400P. As much as I can offload from the host onto the device, the better (will have multiple cameras providing point clouds)

edit: corrected to state device-side point cloud conversion

[Erol444]

jjd9 on Discord got on-device depth->pointcloud working using Kornia's depth_to_3d function, source code here. cc @madgrizzle