search

cvar-upm / cvg_ardrone2_ibvs

Vision based control for object following in UAVs
http://robotics.asu.edu/ardrone2_ibvs/
Other
85 stars 46 forks source link

readme

cvg_ardrone2_ibvs

Safety Warning and Disclaimer

You are using this software at your own risk. The authors decline any responsibility for personal injuries and/or property damage.

The AR Drone 2.0, supported by this framework, is a TOY. However, its operation might cause SERIOUS INJURIES to people around. So, please consider flying in a properly screened or isolated flight area.

Description

We present a vision based control strategy for tracking and following objects using an Unmanned Aerial Vehicle. We have developed an image based visual servoing method that uses only a forward looking camera for tracking and following objects from a multi-rotor UAV, without any dependence on GPS systems. Our proposed method tracks a user specified object continuously while maintaining a fixed distance from the object and also simultaneously keeping it in the center of the image plane. The algorithm is validated using a Parrot AR Drone 2.0 in outdoor conditions while tracking and following people, occlusions and also fast moving objects; showing the robustness of the proposed systems against perturbations and illumination changes. Our experiments show that the system is able to track a great variety of objects present in suburban areas, among others: people, windows, AC machines, cars and plants.

This project is operative and based on ROS. It was publicly demonstrated in the 11th anniversary of the IEEE International Symposium on Safety, Security, and Rescue Robotics, SSRR2013 (Linköping, Sweden).

An explanation of the stack software and connectivity between modules are specified in the following documents and sites:

Table of Contents

Installation

Pre-requirements

This driver has been tested on Linux machines running Ubuntu 14.10 (64 bit). However it should also work on any other mainstream Linux distributions. The driver has been tested on ROS "groovy". The code requires a compiler that is compatible with the C++11 standard. Additional required libraries are: boost and ncurses. To see dependencies upon other ROS package depends on these ROS packages: ardrone_autonomy, opencv2, roscpp, image_transport, sensor_msgs and std_srvs.

Installation Steps

The installation follows the same steps needed usually to compile a self-contained ROS stack. You can also refer /installation/installation_instructions.txt for installing the stack

Coordinate Frames

Multirotor coordinate frame

In the documentation located in ${IBVS_STACK}/documentation/Coordinate_Frames/Coordinate_Frames_documentation.tex/pdf, this reference frame is called F_{drone_LMrT}.

The reference frame that is used to reference the multirotor's telemetry, broadcasted by the multirotor's ROS driver, is:

[Fm] the mobile reference frame is centered on the drone, with:
 [xm] horizontal and pointing forward (same direction as the optical axis of the frontal camera
 [ym] horizontal and pointing rightwards from the center of the drone
 [zm] vertical and pointing downwards

[F] the fixed reference frame:
 [x] horizontal and pointing "North" (where the magnetometer reading finds the North)
 [y] horizontal and pointing "East"
 [z] vertical and pointing downwards (the z coordinate of the drone is, thus, always negative)

[yaw (mobile to fixed)] increases when the drones rotate in clock-wise direction

The sign convention for the commands, received by the multirotor's ROS driver, is the following:

[paraphrased] The yaw, pitch, roll angles commands are related to the above mentioned [Fm] reference frame (thus setting their sign convention). The dz/"gaz" command is such that the AR Drone is commanded to go higher (increased altitude) when dz/"gaz" is positive. It can be thought of as the dz/"gaz" command setting a higher thrust on the propellers. 

[comment] the following phrases are understood from the point of view of a person sitting on the drone looking forward (looking in the same direction as the frontal camera), under no external wind conditions.
[pitch][+] move backwards
[pitch][-] move forward
[roll][+] move rightwards
[roll][-] move leftwards
[dyaw][+][speed command] rotate clockwise (as seen from above), or N > E > S > W > N > ...
[dyaw][-][speed command] rotate counter-clockwise (as seen from above), or N > W > S > E > N > ...
[dz][+][speed command] increase altitude, move upwards
[dz][-][speed command] decrease altitude, move downwards

How to Run

Launch scripts

In order to run the stack, it was decided to run each node in a separate tab of a terminal window. The initialization of the architecture is done by executing shell scripts that open a new terminal with each node running in its tab. The script that is available is the following (please take a look at them to understand how do they work):

NOTE: all the launchfiles open a separate terminal with multiple tabs, where each tab usually runs only one tab. If you close the terminal tabs using the close button at the corner of the window which has multiple tabs, then only one of the tabs will be closed correctly (the one that is currently selected):

The launch scripts have to be called using the following sintax in the shell terminal: 

```bash
$ cd ${IBVS_STACK}/launchers
$ ./parrot_IBVSController_launcher_Release.sh
```

Interaction with UI

Simplified instructions to work with the controller:

  1. Window "OpenTLD GUI":
    1. It is important to familiarize with this window. It allows you to start/stop the tracker, toggle learning, change the tracker's target
    2. To start the tracker:
    3. click F5 to update the image shown on the OpenTLD GUI, you have to do this sometimes because no image is shown on ir, or because you want to track something that is not on the currently shown image
    4. Select the target drawing a bounding box around it in the image
    5. press "Enter"
    6. To change the tracker's target:
    7. click "reset" once
    8. redo the "start the tracker" instructions
    9. To toggle learning:
    10. You have to be tracking an object. Then click the toggle tracking button.
    11. This will change from "learning and tracking mode" to "tracking only mode".
    12. To know exactly whether the tracker is currently learning or not; then go to the terminal tab which is running the OpenTLD nodes. This information is printed there.
  2. To interact with the rest of the software the "IBVSCntInterf" terminal tab is used:
    1. To check the keybindings the best is to read the code on the source file: ${repository}/parrotBrainInterface/src/sources/IBVS_interface.cpp . There is a switch case instruction which acts upon pressed keys.
    2. The important keys are:
      't': take off
      'y': land
      'h': enter hovering mode
      'm': enter move mode
3. In move mode:
      arrow_left: move/tilt left
      arrow_right: move/tilt right
      arrow_up: move/tilt forwards
      arrow_down: move/tilt backwards
      'q': move upwards
      'a': move downwards
      'z': yaw left/counterclockwise
      'x': yaw right/clockwise
      's': stop / send "zero" commands in every command. In ROS ardrone_autonomy this means entering the hovering mode  
4. Start the controller:
      'o': start the controller
      'i': stop the controller
      'u': reset controller (unimportant command with the IBVS controller)
5. Start the kalman filter (only used for data logging):
      'l': start the kalman filter
      'k': stop the kalman filter
      'j': reset kalman filter, reinitialize estiamte to (0,0,0)
6. Note that the commands explained in the move mode act differently when the controller is started. They change the controller references in a logical way (you can check it looking at the HUD window without flying and starting the controller). There are some contradictions yet in this options (like move forward would actually make the drone move further away from the target) and bugs (some events trigger a change of controller references back to default values).
7. The logs are saved in ${IBVS_STACK}/logs/drone_logger

We recommend experimenting with the AR Drone and the tracker separately before running the controller. Usually you can recover control of the drone at any moment by stopping the controller ('i') and entering hovering mode ('h').

License

All distributed software, except the packages listed in the following, are subject to 3-clause BSD license (see the file: LICENSE).

This software stack uses other third-party open-source libraries (some of them are inluded in the soure of the stack as separate packages):

Contributors

Other collaborators

Contact Information

Current CVG (CAR, UPM-CSIC) staff that can give limited support to the usage of this stack are:

NOTE: we know that this documentation is partial. If the stack gets the interest of the community we will continue adding information depending on the questions made by its users.