Facing issue while converting Midas depth to pointcloud

[EhrazImam]

Hi @geturin great work,

I am trying to convert midas depth to pointcloud but unable to i can see subscribe topi where i replaced it with my topic but i am not using orbslam3 instead i am using openvins where i have slam points https://github.com/geturin/OAFD_Monocular/blob/41f98f3fb53c3250187efc55c2893373dc32cfbc/src/depthmap_calibration.py#L113 but while i am changing its not publishing any thing.

Secondly if you have depth why not to use depth directly instead of pointcloud.

thankyou

[geturin]

Hi @EhrazImam

I'm sorry, but the project development didn't take compatibility into account.If you want to use another SLAM algorithm or drone, you may need to make multiple modifications.

about depthmap_calibration.py not work，you should check

1. SLAM point cloud is published on /map frame. https://github.com/geturin/OAFD_Monocular/blob/41f98f3fb53c3250187efc55c2893373dc32cfbc/launch/slam.launch#L24

2.Timestamps of SLAM point cloud and depth map inherit from RGB image. If you're using code from the official Midas repository, there might be some differences https://github.com/geturin/MiDaS/blob/64298b72234cec609c50bbb5d0ec4de649d0586c/ros/midas_cpp/src/main.cpp#L264

3.If there are no issues with points 1 and 2, and your computational speed is slow , you can try increasing the queque size in

https://github.com/geturin/OAFD_Monocular/blob/41f98f3fb53c3250187efc55c2893373dc32cfbc/src/depthmap_calibration.py#L115

Tello can't meet VIO requirements, so you're likely using another drone. You'll need to change all about camara parameters in src/depth2pcd.py

reason for using point clouds was that I initially hoped to filter the point clouds once, but eventually, I gave up on this idea

[EhrazImam]

Dear @geturin thankyou for your quick response i tried following your modification to the code and still i am getting no mssgs in /test_pcd.as In 1. i am not using any launch for running depth to pointcloud i am simply running (python3 depthmap_calibration.py).

2. i made changes where "ros_img.header = self.ros_img.header # Use RGB image header for the depth image" and i am using midas only but not the cpp ones the python one where i converted it into ros.

3. and i am getting 20/22 fps so i havent chaged any thing.

4. in depth@pcd.py i did changed the camera parameters

5. i tried with master orbslam3_ros_wrapper and tested on mono(not mono_imu) and it is publishing slam point but still test_pcd is not publishing anything

For reference i have attached link of the updated python script and rosbag of my setup. https://we.tl/t-hqjz1LeAQL

It will be great if you can assist me just for conversion of depth to pcd.

[geturin]

Dear @EhrazImam Alright, first, let's determine if the error occurred in the TimeSynchronizer or the callback function. You can modify your callback function as below to check if your data is triggering the callback

def callback(image,depth,pcd):
    print("ok")

If it's not triggering, I still recommend that you consider increasing the queue size Try set it to 6

ts = message_filters.TimeSynchronizer([camera_sub, depth_sub,pcd_sub], 6)

[EhrazImam]

Dear @geturin

1. i tried to def callback(image,depth,pcd): print("ok")

   but it is not printing

2. I tried ts = message_filters.TimeSynchronizer([camera_sub, depth_sub,pcd_sub], 6/10) but still nothing changes

3. rosgraph

i have checked rgb and depth are synchronized but slamp/point or map/points are not syncronized with rgb and depth how can i make them syncornized

[geturin]

Dear @EhrazImam

If the callback function is not triggered, from my understanding, there could be only three situations:

1. There's no data in one of the topics.
2. The data timestamp in one of the topics is incorrect.
3. The queue size is too small or the computation delay is too large, leading to inability to find consistent stamp data in the queue.

Even though I believe you've already checked these three points, I can't think of any other reasons besides these.

Maybe you can alternately remove one topic from the three in the TimeSynchronizer to confirm which topic is causing the callback not to be triggered. like

ts = message_filters.TimeSynchronizer([camera_sub, depth_sub], 3)

Also, this project uses a version of ORB_SLAM3 and its wrapper that is not the latest. If you're using the latest version, their handling of timestamps may differ.

[EhrazImam]

Dear @geturin can you assist me directly using midas depth in ego planner as i have already successfully ran it on d435i, but when i am replacing it with midas depth i cannot see any grid map. Can you help me with this as i tried to follow you above stem with the help of slam i was not able to but if you can some how help me with this. Thankyou.

Secondly in midas depth when i am converting depth to pointcloud with open3d i can see the the object it to near even if it is not, i am new to mono depth as before this i have tried stereo depth but not with mono depth.

[geturin]

dear @EhrazImam

I'm sorry, but I don't have experience directly inputting depth maps into the ego planner. Initially, I was using point clouds, so I might not be very clear where the potential issues might be. However, you could check the encoding mode of your depth image.

In some similar projects .that I'm aware of, there are inputting monocular depth prediction images, which you might be able to use as references https://github.com/YongzhouPan/Mono-Navigation https://github.com/surfii3z/jy_master_thesis

Regarding your latter question, it's impossible to restore the absolute depth of pixels from a single image; you can only obtain their relative depth. There's a scale between the relative depth and the real depth