How to detect many obstacles in on e scene ?

[mechaliomar]

How to detect many obstacles in on e scene ? using D435 camera

[MartyG-RealSense]

A Deep Neural Network (DNN) can identify multiple objects simultaneously. The links below are a couple of examples for the 400 Series cameras and RealSense SDK.

https://github.com/twMr7/rscvdnn

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/dnn

There are also links to Python-based DNN example programs here, including one called MobileNet-SSD-RealSense:

https://forums.intel.com/s/question/0D70P000006EQctSAG

[mechaliomar]

Thanks for your reply. I am using DJI onboard sdk, I finished path planning with fixed obstacles and now I would like to use the camera to detect and locate obstacles (x y z position) in real time. I'm confused

Le sam. 30 mars 2019 à 16:57, MartyG-RealSense notifications@github.com a écrit :

A Deep Neural Network (DNN) can identify multiple objects simultanreously. The link below has an example for the 400 Series cameras.

https://github.com/twMr7/rscvdnn

There are also links to Python-based DNN example programs here, including one called MobileNet-SSD-RealSense:

https://forums.intel.com/s/question/0D70P000006EQctSAG

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478222380, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFsIfJOM1oHQHt_5kAzOa0RyLyND8ks5vbycQgaJpZM4cTm60 .

[mechaliomar]

i'm using linux

Le dim. 31 mars 2019 à 12:25, Mechali Omar mechaliomar@gmail.com a écrit :

Thanks for your reply. I am using DJI onboard sdk, I finished path planning with fixed obstacles and now I would like to use the camera to detect and locate obstacles (x y z position) in real time. I'm confused

Le sam. 30 mars 2019 à 16:57, MartyG-RealSense notifications@github.com a écrit :

A Deep Neural Network (DNN) can identify multiple objects simultanreously. The link below has an example for the 400 Series cameras.

https://github.com/twMr7/rscvdnn

There are also links to Python-based DNN example programs here, including one called MobileNet-SSD-RealSense:

https://forums.intel.com/s/question/0D70P000006EQctSAG

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478222380, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFsIfJOM1oHQHt_5kAzOa0RyLyND8ks5vbycQgaJpZM4cTm60 .

[MartyG-RealSense]

If you were using obstacle detection for the purposes of obstacle avoidance, open-source flight control software called DroneCode PX4 that can be used with the RealSense 400 Series cameras could provide that for you. It includes an obstacle avoidance function.

Its setup instructions for RealSense are based around Intel's own 'Aero' drone kit, though you may be able to adapt the instructions for your DJI drone.

https://docs.px4.io/en/complete_vehicles/intel_aero.html

The DroneCode software is distributed as source code on GitHub sites.

https://www.dronecode.org/source_code/

[mechaliomar]

Thank you so much, I'll take a look. I appreciate that a lot☺ BEST REGARDS Mr Mechali Omar UESTC,China

Le dim. 31 mars 2019 à 14:35, MartyG-RealSense notifications@github.com a écrit :

If you were using obstacle detection for the purposes of obstacle avoidance, open-source flight control software called DroneCode PX4 that can be used with the RealSense 400 Series cameras could provide that for you. It includes an obstacle avoidance function.

Its setup instructions for RealSense are based around Intel's own 'Aero' drone kit, though you may be able to adapt the instructions for your DJI drone.

https://docs.px4.io/en/complete_vehicles/intel_aero.html

The DroneCode software is distributed as source code on GitHub sites.

https://www.dronecode.org/source_code/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478316221, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFrtW2vBUnDLacppHl8OW6ttD_Uhbks5vcFdJgaJpZM4cTm60 .

[mechaliomar]

Just to let you know, I'm using C++ for coding

Le lun. 1 avr. 2019 à 08:44, Mechali Omar mechaliomar@gmail.com a écrit :

Thank you so much, I'll take a look. I appreciate that a lot☺ BEST REGARDS Mr Mechali Omar UESTC,China

Le dim. 31 mars 2019 à 14:35, MartyG-RealSense notifications@github.com a écrit :

If you were using obstacle detection for the purposes of obstacle avoidance, open-source flight control software called DroneCode PX4 that can be used with the RealSense 400 Series cameras could provide that for you. It includes an obstacle avoidance function.

Its setup instructions for RealSense are based around Intel's own 'Aero' drone kit, though you may be able to adapt the instructions for your DJI drone.

https://docs.px4.io/en/complete_vehicles/intel_aero.html

The DroneCode software is distributed as source code on GitHub sites.

https://www.dronecode.org/source_code/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478316221, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFrtW2vBUnDLacppHl8OW6ttD_Uhbks5vcFdJgaJpZM4cTm60 .

[mechaliomar]

I have been reading the post. But in my case I don't need to load any map. They are using 'osctomap' for the mapping and 3DVFH+ as local planner. In my case I want to:

• Locate many obstacles in real time as the UAV is moving using Intel D435 depth camera as: ( obs1, obs2....obn);
• Extract x y z position for each obstacle;
• Give those positions to path planner;
• Planning the path;

• Go along the path.

  My problem now is in how to use the camera to detect obstacles. Ihave been reading about U-V disparity and others but still no idea. Best regards

Le lun. 1 avr. 2019 à 08:48, Mechali Omar mechaliomar@gmail.com a écrit :

Just to let you know, I'm using C++ for coding

Le lun. 1 avr. 2019 à 08:44, Mechali Omar mechaliomar@gmail.com a écrit :

Thank you so much, I'll take a look. I appreciate that a lot☺ BEST REGARDS Mr Mechali Omar UESTC,China

Le dim. 31 mars 2019 à 14:35, MartyG-RealSense notifications@github.com a écrit :

If you were using obstacle detection for the purposes of obstacle avoidance, open-source flight control software called DroneCode PX4 that can be used with the RealSense 400 Series cameras could provide that for you. It includes an obstacle avoidance function.

Its setup instructions for RealSense are based around Intel's own 'Aero' drone kit, though you may be able to adapt the instructions for your DJI drone.

https://docs.px4.io/en/complete_vehicles/intel_aero.html

The DroneCode software is distributed as source code on GitHub sites.

https://www.dronecode.org/source_code/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478316221, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFrtW2vBUnDLacppHl8OW6ttD_Uhbks5vcFdJgaJpZM4cTm60 .

[MartyG-RealSense]

With this kind of real-time operation, you naturally want to minimize the amount of processing that needs to be done. A method that comes to mind is a bat-like 'echo location' project called Sonic Sight created for the D435 camera by a RealSense community member.

http://sonic-sight.org/

https://github.com/sonic-sight/sonic-sight?language=en_US

[mechaliomar]

That project is interesting, but I need to try a classic way: -Get stereo videos, -Rectify the video, -Get disparity map in disparity domain, -Get binary map,

I had already get the disparity using D435 camera using the SDK written in C++, What I need now is how to get the binary map using opencv(may be)?

The attached photoa are a similar work but using matlab.

On Mon, 1 Apr 2019, 13:53 MartyG-RealSense, notifications@github.com wrote:

With this kind of real-time operation, you naturally want to minimize the amount of processing that needs to be done. A method that comes to mind is a bat-like 'echo location' project called Sonic Sight created for the D435 camera by a RealSense community member.

http://sonic-sight.org/

https://github.com/sonic-sight/sonic-sight?language=en_US

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478445085, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFoNEwPtGCIrHrOVE_v2Ei3dthS1Xks5vcZ7mgaJpZM4cTm60 .

[MartyG-RealSense]

I could not find attached photos, unfortunately. so I was not sure what you were referring to regarding a binary map.

If you mean saving a binary frame, there is a C++ script in the link below.

https://github.com/IntelRealSense/librealsense/issues/2462#issue-365342545

[mechaliomar]

You can watch this video 'real time obstacles detection and distance estimation in Matlab'

On Mon, 1 Apr 2019, 21:27 Mechali Omar, mechaliomar@gmail.com wrote:

On Mon, 1 Apr 2019, 19:42 MartyG-RealSense, notifications@github.com wrote:

I could not find attached photos, unfortunately. so I was not sure what you were referring to regarding a binary map.

If you mean saving a binary frame, there is a C++ script in the link below.

2462 (comment)

https://github.com/IntelRealSense/librealsense/issues/2462#issue-365342545

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478545312, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFqIbuihQ96lt7fqBF5tcSptZoIwFks5vcfCcgaJpZM4cTm60 .

[MartyG-RealSense]

A binary map seems to be a MATLAB concept, and so it is difficult for me to give advice on creating one. I researched the subject very carefully, and the best solution I could find was a downloadable function for MATLAB called shrinkWrap that adds the ability to create a binary map.

https://uk.mathworks.com/matlabcentral/fileexchange/29175-shrinkwrap

[mechaliomar]

hello, I found this : Introduction

Depth cameras are increasingly popular in building automation, occupancy management, security and access control, enabled by low-cost depth sensors like the OPT8241 and OPT8320 3D Time-of-Flight chipsets from Texas Instruments http://www.ti.com/lsds/ti/sensors/3d-imaging-sensors-overview.page?DCMP=analog_signalchain_mr&HQS=3dtof. A key benefit depth cameras is the ability to use depth to segregate foreground from the background. Once foreground objects are isolated, they can be recognized, tracked and counted using modern image processing algorithms available in OpenCV http://opencv.org/. In this post, I will describe how to use the OPT8241-CDK-EVM depth camera http://www.ti.com/tool/opt8241-cdk-evm, BSD-licensed Voxel SDK https://github.com/3dtof/voxelsdk, and OpenCV http://opencv.org/to create a simple people counter and tracking application.

The general strategy of people counting and tracking is as follow:

1. Foreground-Background Separation
2. Convert to Binary Image and Apply Morphology Filters
3. Shape Analysis
4. Tracking

Le lun. 1 avr. 2019 à 22:10, MartyG-RealSense notifications@github.com a écrit :

A binary map seems to be a MATLAB concept, and so it is difficult for me to give advice on creating one. I researched the subject very carefully, and the best solution I could find was a downloadable function for MATLAB that adds the ability to create a binary map.

https://uk.mathworks.com/matlabcentral/fileexchange/29175-shrinkwrap

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478595164, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFv0uOrgH6R7WqxpRZhRw1ObmPJ6dks5vchNTgaJpZM4cTm60 .

[MartyG-RealSense]

Did the people-counting program that you found meet your needs for detecting obstacles in a scene, please?

[mechaliomar]

Not yet, this is using voxelSdk, I think it's not compatible with Realsense. I'm still looking for it?

On Tue, 2 Apr 2019, 13:52 MartyG-RealSense, notifications@github.com wrote:

Did the people-counting program that you found meet your needs for detecting obstacles in a scene, please?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3629#issuecomment-478854600, or mute the thread https://github.com/notifications/unsubscribe-auth/APKJFo1rshHJqMnvxNDBfDscPNAiSbNUks5vcvAKgaJpZM4cTm60 .

[Jiang1206]

深度神经网络（DNN）可以同时识别多个对象。以下链接是400系列相机和RealSense SDK的几个示例。

https://github.com/twMr7/rscvdnn

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/dnn

此处还有基于Python的的DNN示例程序的链接，包括一个名为MobileNet-SSD-RealSense的程序：

https://forums.intel.com/s/question/0D70P000006EQctSAG

Can the depth module D430 use this DNN?

[MartyG-RealSense]

These DNN programs use RGB to classify the images. The D430 Depth Module does not have an RGB sensor. The only Depth Module (camera boards without a casing) that has an RGB sensor integrated into it is the D415.

---

這些DNN程序使用RGB對圖像進行分類。 D430深度模塊沒有RGB傳感器。 D415是唯一一個集成了RGB傳感器的深度模塊（沒有外殼的攝像板）。

[Jiang1206]

Thank you ，I want to use D430 for object recognition.Are there any references or examples?

[MartyG-RealSense]

@Jiang1206 You can certainly detect objects with the D430's depth sensing. I do not know of a way to do object recognition operations though without RGB. If I were approaching this problem myself, I would add an RGB webcam beside the D430 camera and use that as the RGB stream.

Since the RealSense Viewer program can add ordinary RGB webcams as an RGB source, I would speculate that the RealSense SDK could access an external RGB camera and use that as the RGB source. That may make it practical to use a DNN object recognition example programs such as the ones you linked to earlier.

[RealSenseCustomerSupport]

---

@Jiang1206 There were some discussions about using IR image instead of color image for DNN sample. Please refer to https://github.com/IntelRealSense/librealsense/issues/2972 If you use D430, only Y8 and Y16 format supported for IR, so please configure this accordingly.

[RealSenseCustomerSupport]

---

@Jiang1206 Any other questions? Looking forward to your update. Thanks!

[RealSenseCustomerSupport]

---

@Jiang1206 If no other questions for this ticket, will close it accordingly. Thanks!

[Jiang1206]

Thank you for your answer, I have no problem. @RealSenseCustomerSupport