search

luxonis / depthai

DepthAI Python API utilities, examples, and tutorials.
https://docs.luxonis.com
MIT License
913 stars 231 forks source link

Projection of Depth map into point cloud #180

Open saching13 opened 4 years ago

saching13 commented 4 years ago

Start with the why:

The depth map that we can currently obtain from the DepthAI API is an image that contains depth information at each index of the matrix/image.

Projecting this depth map into point cloud allows us to exploit features like point cloud's registration, subsampling, reconstruction etc.

For the initial implementation, this step is done on the host side instead of the Myriad X

It is important to do this on the Myriad X itself since it can unlock the above-mentioned features to be carried out on the Myriad X and reducing more load on the CPU.

The How:

Considering the depth map image is relative to the right camera of the stereo camera. Using its intrinsic parameters, we can obtain the (x_k, y_k, 1) in the camera reference frame. Multiplying z value from the depth map will provide us with the (x, y, z) of each point captured by the stereo camera.

The what

Support point cloud projection on the DepthAI itself, with options like

julianaddison commented 4 years ago

Hi @saching13,

I'm trying to extract the depth map (i.e. depth_raw stream) alongside the raw RGB data stream. I was hoping you might have some experience with extracting it given your reference to the depth map.

I've managed to save the RGB video stream via the line below and am trying to understand if both RGB and depth_raw streams can be saved similarly. Are you aware of any way I could do this?

python3 test.py -v [path/to/video.h264]

saching13 commented 4 years ago

Currently, I have implemented it with right frame as of now. you can take a look at that here. We will soon release an update with rgb camera also. If you want to combine with rgb right away with good accuracy you need to first get the extrinsic calibration between rgb and right camera of stereo and project the depth_raw frame into the rgb camera's frame.

I am not sure about the way to save the depth_raw. I am adding @Luxonis-Brandon here for more details.

Luxonis-Brandon commented 4 years ago

I think @VanDavv has been working towards this using a PR.

Luxonis-Brandon commented 3 years ago

This is implemented in develop and an example program is here: https://github.com/luxonis/depthai-experiments/tree/master/point-cloud-projection

Luxonis-Brandon commented 3 years ago

Oops, missed the on the Myriad X part... reopening this issue.

John-Dean commented 3 years ago

Is there an estimated timeline for this?

I'm trying to turn multiple cameras into point clouds, merge them together and then perform further code on the result and I'm wondering if I should be looking to do the point cloud generation on the Myriad X (if it's at least 60fps and minimal latency) or on the host PC?

Luxonis-Brandon commented 3 years ago

Hi @John-Dean ,

We're making progress on the point-cloud generation on the Myriad X, but took a detour to make better ROS support, so we haven't finished this yet.

@saching13 can provide more information on this.

But anyway, I would recommend generating the point-cloud on the host for now.

Thanks, Brandon

saching13 commented 3 years ago

Hey @John-Dean , Sorry for the delayed response. I think it is really hard to get point cloud conversion of 60fps on the device side. So Like Brandon suggested it would be better to do it on the host. And on the device will try to minimize the computation as much as possible.

I am thinking best case scenario it would take a month taking into consideration of other tasks, type handling and alignment issues. I have a CPP code here which does it host and exploits multiprocessor. This example converts it to a point cloud and then converts it back into another camera frame. you just need the first part.

saching13 commented 3 years ago

Hey @John-Dean , Sorry for the delayed response. I think it is really hard to get point cloud conversion of 60fps on the device side. So Like Brandon suggested it would be better to do it on the host. And on the device will try to minimize the computation as much as possible.

I am thinking best case scenario it would take a month taking into consideration of other tasks, type handling and alignment issues. I have a CPP code here which does it host and exploits multiprocessor.

saching13 commented 3 years ago

Created an host based example for rgb alignment using gen2. Code can be found here P.S: Follow the instructions in the example carefully on how to modify calibration information manually since calibration api is not available in gen2 yet.

MOj0 commented 2 years ago

Added RGB Point Cloud Visualizer with depth information here.

madgrizzle commented 2 years ago

Is there any chance that device-side point cloud conversion might end up happening, even at a lower FPS and/or resolution? I'm starting to work on ROS2 navigation and think I can get by with something as low as even 15FPS + 400P. As much as I can offload from the host onto the device, the better (will have multiple cameras providing point clouds)

edit: corrected to state device-side point cloud conversion

Erol444 commented 2 years ago

jjd9 on Discord got on-device depth->pointcloud working using Kornia's depth_to_3d function, source code here. cc @madgrizzle image