search

luxonis / depthai-ros

Official ROS Driver for DepthAI Sensors.
MIT License
239 stars 173 forks source link

Update Depth Alignment #325

Closed borongyuan closed 9 months ago

borongyuan commented 1 year ago

There is currently a misalignment issue using setDepthAlign(CameraBoardSocket camera). So I changed to use setDepthAlign(Properties::DepthAlign align). I also created an issue for this in https://github.com/luxonis/depthai-core/issues/840.

In addition, OAK cameras align depth image to DepthAlign::RECTIFIED_RIGHT by default. But for the convention of many applications, including OpenCV, depth image is aligned to the left camera. So I added this part of options, but the default behavior is still align to the right camera.

Serafadam commented 1 year ago

Hi, thanks for the PR! I'll test it out tomorrow.

borongyuan commented 1 year ago

I seem to understand the reason for misalignment. Even if another image is aligned to a real camera, the image of the real camera is distorted. So the images are still not aligned. It is only reasonable to align to the rectified image.

I did a massive update to the OAK driver for RTAB-Map a while ago. The main problem found at the beginning is this misalignment, and OAK images are aligned to the right camera by default. I have dealt with these issues in RTAB-Map, and added other features, including on-device GFTT and SuperPoint support.

But now back in ROS, we still need to deal with these issues. The convention of OpenCV and ROS is to align images to the left camera. Now it seems that camera_info also needs to be modified. The translation part of the left camera's projection matrix need to be 0. Otherwise there are many algorithms that would not be usable.

Serafadam commented 9 months ago

Hi @borongyuan , sorry for the delay, but both depth alignment and OpenCV conventions have been added to latest Humble branch as part of other PRs, for now I will be closing this PR, please reopen/comment it if you have further comments.