Block Matching Disparity Map Package Upgrade

[sah4jpatel]

Requirement: Integrate Block Matching-based Disparity Map for Depth Estimation in Vision Pipeline

Implementation Plan:

• Refer to Summer 2023 stereo vision code base and update code from Foxy to Humble Compatibility
• Develop C++ standalone driver code to run without ROS code to develop and test locally
• Integrate into VIsionModule repo

[sah4jpatel]

Existing codebase very reliant on ROS1 commands for image handling. Working on a minimum example of a ROS2 version and going from there. May need @Rhyme0730's advice on how to deploy and test this.

[sah4jpatel]

Didn't change anything, I was able to somehow get that stereo stuff to compile.

[sah4jpatel]

It works. Was able to calibrate the camera and run the Summer2023 Disparity Map stuff and James said "it looks pretty good."

Awaiting on @Rhyme0730 's vision integration to test the point cloud visualization code

[sah4jpatel]

The issue is that the new cameras are higher resolution and thus slow the process down quite a bit

[sah4jpatel]

Need to remove the noise a bit

[sah4jpatel]

@J4mZzy My update for the daily meeting:

I was able to get James's old code to work it compiles fine. I didn't have to change anything, unsure what Rao was dealing with. For next steps, I'm going to be copying over the important parts of the deprecated code into the vision repo and then from there, I will be cutting back and trying to benchmark that on the OPI to see what the speed is , and then following that I will be trying to make optimizations to see if we can use the RKNNAPI to accelerate the block matching process. The issue is that the I will have to re-implement block matching probably line by line from the raw algorithm and I probably not be able to use the open CV one.

[Rhyme0730]

Sahaj, can you get the disparity work? Such as you can get a specific coordinate’s distance through this ros package. The code that I wrote can get the coordinates of center of object. So I need to get the distance of a specific coordinates (x, y) and then everything can work.

On Thu, Aug 29, 2024 at 19:22 Sahaj Patel @.***> wrote:

@J4mZzy https://github.com/J4mZzy My update for the daily meeting:

I was able to get James's old code to work it compiles fine. I didn't have to change anything, unsure what Rao was dealing with. For next steps, I'm going to be copying over the important parts of the deprecated code into the vision repo and then from there, I will be cutting back and trying to benchmark that on the OPI to see what the speed is , and then following that I will be trying to make optimizations to see if we can use the RKNNAPI to accelerate the block matching process. The issue is that the I will have to re-implement block matching probably line by line from the raw algorithm and I probably not be able to use the open CV one.

— Reply to this email directly, view it on GitHub https://github.com/BUZZ-Blimps/CatchingBlimp/issues/40#issuecomment-2319401857, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5WJZGINMJDCYYBPBDGFRIDZT6UMXAVCNFSM6AAAAABNAV2ILOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJZGQYDCOBVG4 . You are receiving this because you were mentioned.Message ID: @.***>

[sah4jpatel]

I was able to get the disparity map running without changing anything. However, I’m not sure if that means we have numeric values. We will have to ask James.

[J4mZzy]

I was able to get the disparity map running without changing anything. However, I’m not sure if that means we have numeric values. We will have to ask James.

We do have numerical values, we just need to use the bounding box info to filter the distance for each object.

[sah4jpatel]

Update: Benchmarked the OpenCV StereoBM process on the OPi

Average Rectify Time: 18.8547 ms
Average Disparity Time: 29.6001 ms
Average Total Time (excluding capture): 48.4554 ms
FPS: 20.6375

Note this is not with ROS2 overhead, will have that data shortly.

[J4mZzy]

Update:

Benchmarked the OpenCV StereoBM process on the OPi

Average Rectify Time: 18.8547 ms

Average Disparity Time: 29.6001 ms

Average Total Time (excluding capture): 48.4554 ms

FPS: 20.6375

Note this is not with ROS2 overhead, will have that data shortly.

This is very fast!!! Looking forward to see the Rosified speed!

[sah4jpatel]

Getting the 2023 code working on the OrangePi required the following:

Installing rosdep for ROS 2 Humble

If you encounter an issue where the rosdep command is not found while using ROS 2 Humble, it likely means that the rosdep package is not installed. Follow these steps to install rosdep and resolve the issue:

Step 1: Install rosdep

You can install rosdep using the package manager for your operating system. For Ubuntu, use the following command:

Installing rosdep for ROS 2 Humble

If you encounter an issue where the rosdep command is not found while using ROS 2 Humble, it likely means that the rosdep package is not installed. Follow these steps to install rosdep and resolve the issue:

Step 1: Install rosdep

You can install rosdep using the package manager for your operating system. For Ubuntu, use the following command:

sudo apt install python3-rosdep

Step 2: Initialize rosdep

After installing rosdep, you need to initialize it to set up the rosdep database:

sudo rosdep init
rosdep update

Step 3: Verify Installation

To ensure that rosdep is installed correctly, run the following command:

rosdep --version

This command should display the version of rosdep, confirming that it is installed.

Step 4: Usage

Once rosdep is installed and initialized, use it to install dependencies for your ROS 2 workspace:

rosdep install --from-paths src --ignore-src -r -y

By following these steps, you should be able to resolve the issue of the rosdep command not being found and proceed with managing dependencies in your ROS 2 Humble environment.

This is all to solve the following error:

--- stderr: image_proc
/home/opi/GitHub/CatchingBlimp/deprecated/Summer2023/ros2_stereo/src/image_proc/src/rectify.cpp:43:10: fatal error: tracetools_image_pipeline/tracetools.h: No such file or directory
   43 | #include "tracetools_image_pipeline/tracetools.h"

[sah4jpatel]

Getting the old code working on the Pi is causing some funny behavior where the Camera tuns off when the rectification node in image_proc is started in any capacity. Will troubleshoot later

[sah4jpatel]

Have it running on the OPi, I think I wasn't supplying enough power to it with my temporary cable so the camera would power cycle. However, at my apartment the performance is SPF instead of FPS. I will try it again after recalibrating this camera at the lab tomo.

[sah4jpatel]

Created a single-node version of the vision pipeline and it can run at 9-10 FPS on the Orange Pi (not including network delays upon publishing). There may be further optimizations worth doing but I will prioririze validating the designs and achieving feature parity with the old code.

It is the first commit into the VisionModule repo: https://github.com/BUZZ-Blimps/VisionModule/tree/main

[J4mZzy]

LFG!!!!!!

[sah4jpatel]

Added a lite version of multithreading

Laptop Performance:

split_time: 0.0
debay_time: 0.434
rectify_time: 0.967
disparity_time: 34.741
total_time: 36.145
fps: 27.666343892654584

Orange Pi Performance:

split_time: 0.001
debay_time: 1.48
rectify_time: 7.883
disparity_time: 66.824
total_time: 76.19
fps: 13.125082031762698

[sah4jpatel]

graph TD
    A[Image Split] --> B[Image Rectify]
    B --> C[Image Debay]
    C --> D[Disparity Map]
    C --> E[YOLO Inference]
    D --> F[Publish results]
    E --> F

This is the current process for this vision pipeline. YOLO and disparity are parallelized.

The YOLO time is the difference between disparity map ending and YOLO inference ending.

[sah4jpatel]

Next steps, slow and steady optimization to improve this performance

Integrate other depth estimation technique as well, and help integrate disparity map filtering

[sah4jpatel]

YoloV10 is a bit annoyinh to get working with rknn but I have the process down:

1. A model is to be trained using the following repo instead of the normal ultralytics package: https://github.com/airockchip/yolov10/tree/main
2. The resulting model is to be exported using yolo export model=model/yolov10{n/s/m/b/l/x} format=rknn opset=13 simplify
3. This .onnx file is to be converted to an RKNN file which can be done from a docker environment that has rknn_toolkit2 installed
4. The resulting RKNN file has to get special post-processing in order to work as expected but I copied it from the model_zoo

[sah4jpatel]

quantized yolov10n

[sah4jpatel]

performance breakdown of the vision module