Deploy object detection on C/C++ for better performance

[Bill-Haoyu-Lin]

Transfer python opencv and pytorch code to C/C++, in order to get higher fps(lower delay, faster processing time). We wanna try this for faster detection. Currently we have only 20-23 fps with python3 and yolov3 model. (transfer the code in cv.py to C/C++ )

[Bill-Haoyu-Lin]

https://docs.opencv.org/4.x/dd/d55/pytorch_cls_c_tutorial_dnn_conversion.html \ Here is the link to opencv documentation if you need more info.

[Bill-Haoyu-Lin]

We don't need pytorch for C/C++, pytorch is just easier for us to train our model with.

[JustinLinKK]

• [x] Deploy OpenCV on C++ (Linux Version) (Need to be done on the TX2 Board) (Also do it on your computer)
• [x] Deploy function on object detection and tracking. (Easy)
• [ ] Optimize framework processing using CUDA kits (Most difficult but mandatory. Without it, the TX2 is just garbage)
• [ ] Import Motion prediction function (Not emergent task)
• [ ] Priority selection function for motion prediction

[JustinLinKK]

Mainly use the visual studio to do this subproject. The environment would use CMake and OpenCV, which is easier to deploy in Linux. It needed to be tested under a virtual machine before deployed https://learn.microsoft.com/zh-cn/cpp/build/get-started-linux-cmake?view=msvc-170 CN guidance https://learn.microsoft.com/en-us/cpp/build/get-started-linux-cmake?view=msvc-170 EN guidance

[Bill-Haoyu-Lin]

Mainly use the visual studio to do this subproject. The environment would use CMake and OpenCV, which is easier to deploy in Linux. It needed to be tested under a virtual machine before deployed https://learn.microsoft.com/zh-cn/cpp/build/get-started-linux-cmake?view=msvc-170 CN guidance https://learn.microsoft.com/en-us/cpp/build/get-started-linux-cmake?view=msvc-170 EN guidance

You can use WSL on ur device for linux system deploy. You should already have it with CVAT setup @JustinLinKK

[JustinLinKK]

The initial environment is set up based on Cmake and OpenCV. Now working on object detection code translation.

[JustinLinKK]

I had a meeting with my friends working on AI last night. Qualcomm friends say PyTorch is more recommended than TensorFlow in this environment. The performance difference between Python and C++ is negligible (1-2fps diff). I would keep coding the C++ using the PyTorch library, currently processing to Model input. Most of the camera capture and image show is finished, and I am working on the AI model. Additionally, the framework preprocesses using OpenCV could be further development features (Next Generation). The standard process is to export the Red channel to ignore friendly. Return it as a black-white RGB framework and process it with a tiny Yolo model. Or replace the first detection convolutional layer of yolo with the one-channel layer. But the performance difference is needed to be tested. Generally, the prediction of that is the improvement because of the complexity reduction of the CNN.

[Bill-Haoyu-Lin]

I had a meeting with my friends working on AI last night. Qualcomm friends say PyTorch is more recommended than TensorFlow in this environment. The performance difference between Python and C++ is negligible (1-2fps diff). I would keep coding the C++ using the PyTorch library, currently processing to Model input. Most of the camera capture and image show is finished, and I am working on the AI model. Additionally, the framework preprocesses using OpenCV could be further development features (Next Generation). The standard process is to export the Red channel to ignore friendly. Return it as a black-white RGB framework and process it with a tiny Yolo model. Or replace the first detection convolutional layer of yolo with the one-channel layer. But the performance difference is needed to be tested. Generally, the prediction of that is the improvement because of the complexity reduction of the CNN.

Reducing the complexity of RCNN, and CV development is what we have been doing for past month and now. Our solution for enemy differentiation is to use a power switch to indicate what team we are so we just shoot the other team's target by having 2 color armor plate in our model. As people in Qualcomm is using binary files for ML acceleration, a faster rate of detection must be there with different language.

[JustinLinKK]

https://pytorch.org/tutorials/advanced/cpp_extension.html

[Bill-Haoyu-Lin]

can you have the code written and upload to a PR? maybe an extra folder can be used or even extra repo can be made if the code base is too big

[JustinLinKK]

OK, Finally I made some progress on the Cmake project; the version difference on the torch library causes the current difficulty. The model we trained was based on the old versions, and we cannot use it because the code is based on the newest torch version. XD Here is the reference. BTW, the CMake supported the torch library and OpenCV well. Two demos with Video Capture and tensor creation are coded and run successfully. The Code of object tracking is also finished based on the CPU. The current development needs the pt file produced by the newest PyTorch version. PS: The environment setup is pretty painful... Being chased by someone, lmao.

[Bill-Haoyu-Lin]

I will help for transfer the old pytorch model to new pytorch model

[Bill-Haoyu-Lin]

What is your version of libtorch @JustinLinKK

[Bill-Haoyu-Lin]

Need to convert pt model to script for C++ to use on libtorch

[Bill-Haoyu-Lin]

https://drive.google.com/file/d/13XXbswMJKOmh8BOAkDDSyPNEZVvsCUTJ/view?usp=sharing