Noisy depth image from ros driver

[ghost]

HW Info

Camera: OAK-D-Pro-W ROS2 Version: Humble

Depth Image published by the depthai_ros_driver's camera node is very noisy when compared to the depth image shown by running the demo app

Sample, from depthai_demo.py:

Sample, from depthai-ros-driver:

Params kept consistent in both pipelines:

Median Filter (5x5) LR check: True LR Threshold: 10 Resolution: 1280x800 (L+R) Extended Disparity: False LaserDot Brightness: 500 IR FloodLight: 0

Things I've tried:

Enabling Threshold filter: Improves the quality but not by much

Enabling Spatial/Temporal filters: Improves quality slightly but introduces high latency

Changing LaserDot Brightness and enabling extended disparity: No change obsereved

Changing resolution/Median FIler Kernal size: No changes observed

[Serafadam]

Hi, can you post images from left/right camera as well? It can be looking at a simple white wall

[ghost]

Hi @Serafadam,

Thanks for your response, great work on the driver btw!

Here is a snapshot of the Depth, RIght and Left topics from the ros driver:

And a snapshot in the same position (approximately) from the python api:

As you can see the python api is less noisy and note that I have used depth_preview.py which is not doing any post-processing on the host (or the device as far as I can tell).

Another note is that I have not modified the default params in the python API so I am pretty sure the lase brightness is 0 there. So I have tried launching the ros node with the brightness being 0 as well as 500 and 800; Observed no visible improvements or changes.

[ghost]

A quick update, I tried the same test on an OAK-D-LITE to eliminate the possibility of any hardware issues. It does seem to provide a slightly better depth image but when compared to the API, it does seem to be very noisy.

ROS Driver:

API:

[Serafadam]

Thanks for the information, could you also provide topic output when running stereo_inertial_node from depthai_examples?

[ghost]

Which topic(s) do you need me to capture? and would you like a bag?

[Serafadam]

Same, depth output, bags should be fine. You can also enable additional logs with export DEPTHAI_DEBUG=1

[ghost]

Here are the sanpshots, stereo_inertial_node:

API:

stereo_inertial_node:

[Serafadam]

Hm, that's a little bit weird. Could you place the camera in a fixed position, enable debug logging and post debug logs and stereo images (or rosbags) from:

• python example
• stereo_inertial_node
• camera.launch.py
• rgbd_pcl.launch.py

[ghost]

Here are the logs and images:

Camera.launch.py:

Image:

Log: camera_launch_2.log

stereo_inertial_node:

Image:

Log: stereo_inertial_launch.log

rgbd_pcl.launch.py

Image:

Log: rgbd_pcl.log

Python API:

Image: Log: python_api.log

[Serafadam]

Thanks for the logs, I'll try to come back at you starting next week after I go through all of them

[ghost]

Hi @Serafadam, any updates on this issue?

[Serafadam]

Hi, default config for camera.yaml has been updated a little bit, you can see if it helps. I've looked at the logs but unfortunately it's a little bit hard to analyze them since Node properties are obscured. You can try adding following lines:

    std::ofstream file("/tmp/pipeline.json");
    file << pipeline->serializeToJson()["pipeline"];

after this log line, for python add

        with open("/tmp/pipeline.json", 'w') as f:
            json.dump(self._pm.pipeline.serializeToJson()["pipeline"], f)

after this line.

[ghost]

Hi @Serafadam,

Here are the outputs and the json logs,

Demo App:

Log: pipeline_demo.json.log

ROS Driver:

Log: pipeline_ros.json.log

[Serafadam]

Hi, thanks! Notes based on those logs:

1. It seems that in ROS depth is not aligned to RGB, is that on purpose? Default config values set RGBD alignment
2. LRC Threshold is set to 10 in ROS, 4 in Python, although I would stick to 10.
3. Confidence threshold is set to 245 in python, also shouldn't make a difference
4. Threshold filter values are different but it's disabled in both pipelines anyway
5. This one is a bigger difference - Mono Camera resolutions are set to 400 in python example and 800 in ROS-although it might seem that bigger is better, increased resolution might produce more noise depending on environment.

So, to sum up, I would try changing two parameters and see if we get the same results:

• stereo.i_align_depth to true
• left.i_resolution to 400
• right.i_resolution to 400 Let me know if that helps. Also, please note that visualizations of depth streams might differ due to how data is interpreted by the renderer.

[ghost]

Hi, so the align_depth is purposely set to false since I will be using the Depth pipeline instead of RGBD to reduce bandwidth over USB. This should not affect the noise correct?

Reducing the resolution on the ROS does seem to improve the noise a little bit but not by much, I will test this a few more times to confirm.

And about the rendering, I did consider this, and the problem is not visual since the point cloud that is generated from the depth image is extremely noisy as well. It could be the case that the visualisation in the SDK is different and is "hiding" the noise but at the end of the day I need the noise on the ROS side to be minimal and that is the only reason for the comparison.

So apart from the resolution and depth alignment, there is nothing else that can be done to reduce the noise?

Quick note: I did notice that the streams are not rectified in the ros_driver but they seem to be in the stereo_inertial_node. And i cannot find which param to set in the camera.yaml file to set the rectification flag. So please let me know about this as well. Thanks!

[Serafadam]

Hi! Sorry for the delay Regarding noise, I would try playing with available filters via config files. Regarding alignment, different perspective might cause some distortions, though we'll be adding additional filtering for that in near future. Additionally, you can enable subpixel mode in newest humble release if you don't use NNs at the same time (this is a bug occuring with depthai v2.19, it's fixed in newest release). Additionally, there is a WLS based filtering node in progress that should be added soon. Regarding rectified streams from stereo, it's in the backlog and will be added probably at the same time as WLS filter - those functionalities are already coded in, but they've been postponed due to other tasks taking priority.

[RemiFabre]

@AJsamk did you fix your issue? I believe I have the same situation. I'm using an OAK-D, the outputs for the depthai_demo.py are clean, but I get a lot of noise when using the ROS driver.

This also seems linked to #214. For reference, my point cloud looks like this:

Best,

[Serafadam]

Hi, are you using default config files? Regarding previous post - we've recently updated our docs on depth tuning, you can find more information here, hope that helps. WLS filter feature was delayed a little bit, but soon it's going to be added as part of new depthai_filters package

[RemiFabre]

Hi, Yes, I'm using the default config files. I'll update this thread if I have an improvement

[Serafadam]

Hi, WLS filter example has been added in https://github.com/luxonis/depthai-ros/pull/247

[ghost]

HI @Serafadam,

Thanks for the update, I tried the new PR, it does look a lot better in the stereo image as well as the filtered image, although the filtered image is too dark to see, the noise does seem improved quite a bit.

I will try and pipe the new images through to the pointcloud, since I don't see any pcl messages being published I am assuming the filter example does not add that component. Hopefully there will be low to no noise in the pcl.

[LanWu076]

Hi @Serafadam,

Thank you for the updates. I have tried the latest noetic branch (not the 271_neotic). It seems the same issue remaining in the noetic version. The point cloud is okay when running the depthai_example but extremely noisy when running the depthai_ros_driver.

depthai_example (side view) it is okay

depthai_ros_driver(side view) it is noisy

depthai_example (top view) it is okay

depthai_ros_driver(top view) it is noisy, even travel through the wall

I tried to enable the filtering based on the default config file (camera.yaml). It does not affect the performance.

[roboav8r]

Hi all, I'm getting a similar issue when running camera.launch.py. The demo script looks like this: But the output from camera.launch.py when running RGBD and mobilenet looks like this:

I first noticed this because my detections were incorrect, and found this issue as a possible culprit. For example, I was sitting at (0,0,8) but /oak/nn/spatial_detections/.../position was showing (-2.9, 1.2, 6.0). I tried:

• swapping cameras
• running yolo/mobilenet
• examining the inNetData->detections[i].spatialCoordinates data that is going into the SpatialDetectionConverter. spatialCoordinates are incorrect going into the ROS message converter, so whatever is causing the issue is upstream from there
• depthai_examples/launch/yolov4_publisher.launch.py seemed to give reasonable results for spatial detections

Happy to help find the cause, @Serafadam please let me know if there is anything I can do.

[Serafadam]

Hi @roboav8r, one of the changes regarding ROS camera is subpixel mode, you can enable it by setting stereo.i_subpixel: true parameter. Regarding detection results, could you try with setting nn.i_disable_resize: true and setting the rgb.i_preview_size to one supported by the model?

[roboav8r]

Thanks @Serafadam! Looks like that did the trick and it is now outputting reasonable results.

If anyone else is curious, here is my configuration:

/oak:
  ros__parameters:
    camera:
      i_enable_imu: true
      i_enable_ir: true
      i_nn_type: spatial
      i_pipeline_type: RGBD
    stereo:
      i_subpixel: true
    nn:
      i_nn_config_path: depthai_ros_driver/yolo
      i_disable_resize: true
    rgb:
      i_preview_size: 416

[flabrosse]

Can the stereo depth confidence threshold be changed using the ROS driver? I can't see any option that would do that.