Using Computer Vision for ArduPilot/Drone Autonomous navigation

[mtduman]

Hi Steve,

I am working as a Computer Vision Engineer and have some question regarding Drone.

My company is an Engineering/Consulting firm and has many drones for Bridge Inspection. On the inspection, when the drone are under the bridge (GPS-denied area), they are having problems. I am looking to find a solution for that, such as, if the drone can’t receive the GPS coordinates, can we use stereo cameras/Computer Vision to avoid drift and make the drone autonomously navigate to the next waypoints.

I did my homework and watched all Redtail videos and tried to read all related documents/GitHub. Also did research about IMU and VISLAM, and many other related subjects.

Could you please help me about how I can add autonomous navigation feature on GPS-denied areas by using NVIDIA Jetson devices and Computer Vision on a drone.

It looks like ZED 2 stereo cameras have many great futures to help autonomous navigation. Does ZED 2 cameras could be a solution for that or you have any other suggestions?

Thank you in advance.

Best, Matt

[mtbsteve]

Matt, thanks for your comments. The most straightforward solution would be using an optical flow sensor. That allows you to position your drone in an GPS denied environment and to navigate by waypoints in Guided mode. Redtail is using such a sensor for example. For visual navigation, there are a number of projects underway, besides Redtail, this one might be of interest to you: https://discuss.ardupilot.org/t/gsoc-2020-enhancements-of-non-gps-navigation-and-object-avoidance-with-realsense-cameras/55984

For a commercial grade solution for your enterprise, you may also look at Skydio, their Skydio2 drone is likely the most advanced solution for visual navigation on the market today, and is actually based on the Nvidia TX2.

[mtduman]

Great information! Many thanks, Steve

[mtduman]

Hi Steve, I have more question than before. I looked to the GSoC project in detail. If we add stand-alone tracking camera and depth camera to our drone, such as Intel RealSense T265 tracking camera and Intel Depth Camera (or all in one ZED2 Tracking+Depth Camera), for autonomous navigation on GPS-denied environment, we can navigate autonomously. Could you please tell me, what are the possible problems and challenges we will face on this navigation?

Related with that, I have these questions on my mind; How far can we autonomously navigate, 20/30 feet or more?
Can we navigate securely just using depth camera input? Do we need light for cameras to capture environment clearly? Should we expect any drift problem? Thanks in advance!

[mtbsteve]

Could you please tell me, what are the possible problems and challenges we will face on this navigation?

Plenty. The biggest challenge will be to write the code and to train the DNNs needed. You need to define first and foremost what the drone should do exactly. Is it to detect obstacles? Or also to navigate around obstacles? Is it to find a path forward autonomously? Is it to navigate along pre-defined waypoints? Are there specific patterns to follow, eg in the case of Project Redtail, a trail? And so on...

How far can we autonomously navigate, 20/30 feet or more?

In theory yes. But that depends on the overall complexity of the task you want to accomplish.

Can we navigate securely just using depth camera input?

Yes. See the Skydio demo videos for example.

Do we need light for cameras to capture environment clearly?

Yes. As with any computer vision projects, you need appropriate light settings in order to work.

Should we expect any drift problem?

Yes. If you rely purely on depth images, you will experience some (significant) drift. You may compensate this with eg optical flow sensors, but even then you will experience some drift in the range of a couple of feet depending on the surrounding surface and light conditions. Another solution may be to use apriltags. https://ardupilot.org/dev/docs/ros-apriltag-detection.html