Unable to control multiple drones on independent schedules

[amaalo2]

I would like to have one drone going about its own independent path (say it is making a vertical oscillation) and the other drone to move using an obstacle avoidance strategy. What I am seeing, is that currently the architecture can handle two drones at the same time but only when their movements are synched together.

For example,say I have two drones moving at the same velocity. drone_vertical would go +5k and at the same time drone_2 would move +3i. Only once both movements are completed would drone_vertical move say -5k and drone_2 -4j.

What I want is to have drone_2 initiate right away the -4j command without waiting for the drone_vertical to achieve the height of +5k.

Does Airsim currently have the ability to achieve this feature?

@lovettchris

[xxEoD2242]

Hey @amaalo2,

Let me make sure I understand what you want. You want for drone 1 to fly up and around, performing a loiter while another drone moves on the x-axis and does some sort of maneuver, right?

If this is the case, then what you can do is just never call join on any of the movement commands. For example, this is a method that I use for moving drones on different coordinates, when doing drone swarm work.

def fly_to_new_positions(client, vehicle_names: list, new_positions: list, vehicle_offsets: dict, together_tracker: list, stop_matrix: list) -> None:
    for i, name in enumerate(vehicle_names):
        new_position = new_positions[i]
    client.moveToPositionAsync(new_position[0], new_position[1], new_position[2], new_position[3], vehicle_name=name)
    time.sleep(0.1)

I call a sleep because I am sometimes commanding 20+ drones and when you make async calls in Python, you can sometimes overwhelm the server port that is listening for these calls in Unreal Engine.

Now, for what you would like, you can do a few different things.

1. You could define a method that calculates a loiter pattern, maybe an ellipse, with specific waypoints based upon GPS and/or XYZ coordinates.
2. You could have the loiter drone execute something like "Go up 5 meters", wait until the drone is in the loiter position, then execute a movement by velocity that makes it go in a circle/ellipse/etc.

The challenge with all of this is that you have to make sure that the drone that is loitering is going to the correct set of coordinates relative to that particular drone. You have to transform the coordinates if you start the drone NOT at (0,0,0). As for the other drone, you can basically just keep telling it what to do or let it run off in a direction and as long as the method is asynchronous, then the movements won't need to be synced up.

I have read (but not tried) that you can actually connect to two different drones using two separate running threads (python programs) running simultaneously. This way, you can grab control of each vehicle from a different running process and do what you need to do. I would also recommend looking at Python's subprocess.Popen() command, as you can run multiple processes (in this case drones) all from one process. I would recommend scouting the closed issues for AirSim, as I think people have detailed how to accomplish this.

[GeneLYJ]

Hi, I came across this thread while having this question. I have tried using time.sleep() instead of .join() in each iteration in a loop for multiple drones. When the 1st iteration calls Drone_1 to fly a particular position and delay for a second, move to the 2nd iteration, Drone_1 will stop moving and Drone_2 will begin to move. This goes the same for consecutive iterations.

[xxEoD2242]

Can you post your code and settings.json file, please?

[GeneLYJ]

Here's my settings.json file

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
    "ClockSpeed": 1,
  "SimMode": "Multirotor",
  "ViewMode": "Manual",
    "Vehicles": {

        "Drone1": {
            "VehicleType": "SimpleFlight",
            "X": 0, "Y": 0, "Z": -1, 
            "Yaw": 90
        },
        "Drone2": {
            "VehicleType": "SimpleFlight",
            "X": 3, "Y": 0, "Z": -1, 
            "Yaw": 90
        },
        "Drone3": {
            "VehicleType": "SimpleFlight",
            "X": 6, "Y": 0, "Z": -1, 
            "Yaw": 90
        },
        "Drone4": {
            "VehicleType": "SimpleFlight",
            "X": 9, "Y": 0, "Z": -1, 
            "Yaw": 90
        },
        "Drone5": {
            "VehicleType": "SimpleFlight",
            "X": 12, "Y": 0, "Z": -1, 
            "Yaw": 90
        }

    },
    "Recording": {
    "RecordInterval": 1,
    "Cameras": [
        { "CameraName": "0", "ImageType": 0, "PixelsAsFloat": false, "Compress": true }
    ]
}
}

My code

import setup_path 
import airsim

import numpy as np
import os
import tempfile
import pprint
import cv2

import sys
import csv
import time
import keyboard
import math

drone = ["Drone1", "Drone2", "Drone3", "Drone4", "Drone5"]
count = 1
client = airsim.MultirotorClient()
client.confirmConnection()

for x in drone:
    client.enableApiControl(True, x)
    client.armDisarm(True, x)
    time.sleep(1)

for x in drone:
    client.takeoffAsync(vehicle_name=x)
    time.sleep(1)

for x in drone:
    client.landAsync(vehicle_name=x)
    time.sleep(1)

count = 1

Sorry right now even trying to take off it's a problem for me. Like you have mentioned before that, the server ports have to listen calls from Unreal Engine and it causes a bit sluggish during that moment. If this problem is solved, I want to try out making each drones to fly on their perspective paths (all drones to fly at the same time) while collecting all their states. Is this feature able to do without an issue? Thank you.

[amaalo2]

@xxEoD2242 , I tried your first solution and I am still having trouble getting the drone to work asynchronously. I cannot get the moveToPositionAsync API to work unless I have a .join() command. Otherwise, what I see on my terminal is that the script has rushed through the commands and on the simulation no drone movement has been observed.

What I want to do is even simpler. I want drone 1 to move +15k, -15k, +15k and -15k to retrieve its original position. I want drone 2 to move -2j, +2j, -2j, and +2j to retrieve its original position.

I am aware that each drone has its own coordinate frame, and I understand that the unit vector mentioned above are with respect to the local coordinate frame of each drone, but nevertheless the commands that I send are always synched up.

I set up a script to see if two drones can work on independent schedules, and what I observed is not good. When the first waypoint is called, drone 1 reaches ~13 m and does not reach the target of 25m. In that time, drone2 has reached the first waypoint, and then the script jumps to the next waypoint.

Below you can find the python script that I use

import setup_path import airsim import pprint import time

def connect(vehicle_name): client = airsim.MultirotorClient() client.simPause(True) client.confirmConnection() client.enableApiControl(True, vehicle_name) client.armDisarm(True, vehicle_name) client.takeoffAsync(vehicle_name=vehicle_name)

position = airsim.Vector3r(0 , 0, -10)
heading = airsim.utils.to_quaternion(0, 0, 0)
pose = airsim.Pose(position, heading)
client.simSetVehiclePose(pose, False, vehicle_name=vehicle_name)

client.simPause(False)
client.moveByVelocityZAsync(0, 0, -10, 1, vehicle_name=vehicle_name).join()

#Connect the drones connect('Drone1') connect('Drone2')

#Set the waypoint list client = airsim.MultirotorClient() waypoints_drone1 = [[0,0,-25],[0,0,-10],[0,0,-25],[0,0,-10]] waypoints_drone2 = [[0,-2,-10],[0,0,-10],[0,-2,-10], [0,0,-10]]

#Fly the drones for i in range (4): client.moveToPositionAsync(waypoints_drone1[i][0],waypoints_drone1[i][1],waypoints_drone1[i][2],1,vehicle_name='Drone1')#.join() time.sleep(0.1) client.moveToPositionAsync(waypoints_drone2[i][0],waypoints_drone2[i][1],waypoints_drone2[i][2],1,vehicle_name='Drone2').join()

state1 = client.simGetGroundTruthKinematics(vehicle_name="Drone1").position
s = pprint.pformat(state1)
print("state: %s" % s)

airsim.MultirotorClient().reset() client.enableApiControl(False, "Drone1") client.enableApiControl(False, "Drone2")

Settings.json

`{ "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md", "SettingsVersion": 1.2, "SimMode": "Multirotor", "ClockSpeed": 1, "ViewMode": "Manual",

"Vehicles": {
    "Drone1": {
      "VehicleType": "SimpleFlight",
      "X": 0, "Y": 0, "Z": 0
    },
    "Drone2": {
      "VehicleType": "SimpleFlight",
      "X": 4, "Y": 10, "Z": 0
    }

}

}`

[xxEoD2242]

So, the only difference that I see between how you do this and how I do this is that you call a new client, client = airsim.MultirotorClient() for each drone. Then when you want to move the drones, you call an additional client. I'm not super familiar with how the client works in AirSim, but I would suggest defining one client at the very top, and then pass in the the variable into your connect method. Delete the client on line 32 and see what happens.

I have some theories on why this happens but I need to go look at the code to determine how the client object tracks each process.

[amaalo2]

So I did put the client = airsim.MultirotorClient() into my connect(client, vehicle_name) method, deleted the client on line 32 and I still have the same issue.

I also tried to run the simulation without the .join() and using time.sleep() but the problem is time.sleep() imposes a global sleep and not local to each drone. I renamed lines 33 to waypoints_drone2 and 34 to waypoints_drone1 and reran the same script. What I am observing is that the .join() finds the time required to execute the specific command, so in this case moving drone2. Since it takes more time to execute the command for drone2 then drone1 just idles, whereas running the script above as is would interrupt drone1 before it can complete its task.

The only way I see this working in the current architecture is sending parallel commends in different scripts, but I do not know if this doable.

[amaalo2]

So after more experimentation, I was able to find a workaround, but I would still like to keep the git issue open. The idea was to split the code for drone1 and drone2 into two scripts, and have two terminals running each script separately. In that way the drones were able to run on separate threads and have each process run independently as required.

Here is the code I used

Drone1 :

import setup_path import airsim import pprint import time

client = airsim.MultirotorClient() def connect(client, vehicle_name): #client = airsim.MultirotorClient() client.simPause(True) client.confirmConnection() client.enableApiControl(True, vehicle_name) client.armDisarm(True, vehicle_name) client.takeoffAsync(vehicle_name=vehicle_name)

position = airsim.Vector3r(0 , 0, -10) heading = airsim.utils.to_quaternion(0, 0, 0) pose = airsim.Pose(position, heading) client.simSetVehiclePose(pose, False, vehicle_name=vehicle_name)

client.simPause(False) client.moveByVelocityZAsync(0, 0, -10, 1, vehicle_name=vehicle_name).join()

#Connect the drones connect(client,'Drone1')

#Set the waypoint list

waypoints_drone1 = [[0,-2,-10],[0,0,-10],[0,-2,-10], [0,0,-10]]

#Fly the drones for i in range (4): client.moveToPositionAsync(waypoints_drone1[i][0],waypoints_drone1[i][1],waypoints_drone1[i][2],1,vehicle_name='Drone1').join()

state1 = client.simGetGroundTruthKinematics(vehicle_name="Drone1").position
s = pprint.pformat(state1)
print("state: %s" % s)

client.enableApiControl(False, "Drone1")

Drone2

import setup_path import airsim import pprint import time

client = airsim.MultirotorClient() def connect(client, vehicle_name): #client = airsim.MultirotorClient() client.simPause(True) client.confirmConnection() client.enableApiControl(True, vehicle_name) client.armDisarm(True, vehicle_name) client.takeoffAsync(vehicle_name=vehicle_name)

position = airsim.Vector3r(0 , 0, -10) heading = airsim.utils.to_quaternion(0, 0, 0) pose = airsim.Pose(position, heading) client.simSetVehiclePose(pose, False, vehicle_name=vehicle_name)

client.simPause(False) client.moveByVelocityZAsync(0, 0, -10, 1, vehicle_name=vehicle_name).join()

#Connect the drones connect(client,'Drone2')

#Set the waypoint list

waypoints_drone2 = [[5,5,-10],[0,0,-10], [5,5,-10], [0,0,-10]]

#Fly the drones for i in range (4): client.moveToPositionAsync(waypoints_drone2[i][0],waypoints_drone2[i][1],waypoints_drone2[i][2],1,vehicle_name='Drone2').join()

state1 = client.simGetGroundTruthKinematics(vehicle_name="Drone2").position
s = pprint.pformat(state1)
print("state: %s" % s)

client.enableApiControl(False, "Drone2")

[xxEoD2242]

What version of AirSim are you running?

[amaalo2]

According to my settings.json I am using version 1.2

[GeneLYJ]

If you were to run 20 drones then you have to run 20 scripts. That would be a pretty tedious.

[amaalo2]

Exactly, that is why I called my solution above a workaround, and I keep the git issue open.

[xxEoD2242]

Here is my code, you're welcome to run it using AirSim. I tested it out to make sure I wasn't seeing the same issues.

import setup_path 
import airsim

import numpy as np
import pprint
import time
import traceback

def enable_control(client, vehicle_names: list) -> None:
    for vehicle in vehicle_names:   
        client.enableApiControl(True, vehicle)
        client.armDisarm(True, vehicle)

def disable_control(client, vehicle_names: list) -> None:
    for vehicle in vehicle_names:                                                      
        client.armDisarm(False, vehicle)
        client.enableApiControl(False, vehicle)

def takeoff(client, vehicle_names: list) -> None:
    """
       Make all vehicles takeoff, one at a time and return the
       pointer for the last vehicle takeoff to ensure we wait for
       all drones  
    """
    vehicle_pointers = []
    for vehicle in vehicle_names:
        vehicle_pointers.append(client.takeoffAsync(vehicle_name=vehicle))
    # All of this happens asynchronously. Hold the program until the last vehicle
    # finishes taking off.
    return vehicle_pointers[-1]

# ====================================================================================================== #
# Start of main process
# ====================================================================================================== #

# Generate a set of drones based upon a given number input and number of swarms.
# Convention: Capital Letter = Drone Swarm Number = Number of drone in that swarm
# Ex: A1, A2, A3, etc.
# Load list of parameters into the system -> Some sort of class module to set all of these for me.

# Load vehicle names as a list for easy iteration.
# TO DO: This will be drawn from the parameters file loading (Rules sheet)
vehicle_names = ["A", "B", "C"]
vehicle_offsets = {"A": [1, -1, 1], "B": [3, -3, 2], "C": [5, -5, 3]}
time_step = 5 # seconds
final_separation_distance = 10 # meters

# We want a matrix to track who can communicate with who!
# It should be a nxn matrix, with each drone tracking itself and the matrix looks like
#            drone_1 drone_2 ... drone n
# drone_1    true    false   ... true
# drone_2    false   true    ... true
# drone_n    false   false   ... true  
communications_tracker = np.zeros((len(vehicle_names),len(vehicle_names)), dtype=bool)

# We mimic the memory bank of a drone, tracking the relative positions.
# It should be a n-length vector, with each drone tracking itself and the matrix looks like

# drone_1 drone_2 ... drone n
# [x,y,z] [x,y,z] ... [x,y,z]

position_tracker = np.zeros((len(vehicle_names)), dtype=list)

try:
    client = airsim.MultirotorClient()
    print(client)
    client.confirmConnection()
    print('Connection Confirmed')

    enable_control(client, vehicle_names)

    airsim.wait_key('Press any key to takeoff')
    last_vehicle_pointer = takeoff(client, vehicle_names)
    # We wait until the last drone is off the ground
    last_vehicle_pointer.join()

    airsim.wait_key('Press any key to rendevous the drones!')

    vehicle_name = 'A'
    client.moveToPositionAsync(vehicle_offsets[vehicle_name][0] + 5, vehicle_offsets[vehicle_name][0] + 0, vehicle_offsets[vehicle_name][0] + 2, 2, vehicle_name=vehicle_name)
    time.sleep(0.1)
    vehicle_name = 'B'
    client.moveToPositionAsync(vehicle_offsets[vehicle_name][0] - 2, vehicle_offsets[vehicle_name][0] + 1, vehicle_offsets[vehicle_name][0] + 0, 2, vehicle_name=vehicle_name)
    time.sleep(0.1)
    vehicle_name = 'C'
    client.moveToPositionAsync(vehicle_offsets[vehicle_name][0] + 3, vehicle_offsets[vehicle_name][0] + 2, vehicle_offsets[vehicle_name][0] + 0, 2, vehicle_name=vehicle_name)
    # time.sleep(0.1)

    airsim.wait_key('Press any key to reset to original state')
    client.reset()
except Exception:
    traceback.print_exc()
finally:
    if client:
        client.reset()
        disable_control(client, vehicle_names)
    print("Finished!")

Settings.json

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "ClockSpeed": 1,

  "Vehicles": {
    "A": {
      "VehicleType": "SimpleFlight",
      "X": 1,
      "Y": -1,
      "Z": -1,
      "yaw": 180
    },
    "B": {
      "VehicleType": "SimpleFlight",
      "X": 3,
      "Y": -3,
      "Z": -2,
      "yaw": 180
    },
    "C": {
      "VehicleType": "SimpleFlight",
      "X": 5,
      "Y": -5,
      "Z": -3,
      "yaw": 180
    }
  }
}

I didn't catch this before but I think I understand what you are saying and this may or may not resolve your issue. I didn't realize that you were still using the .join() command. I had similar issues as well when using that command and if you really concerned about your drones getting to that position before executing any other commands, then your script idea works great, but is definitely not pretty.

What I do is ignore the fact that drones need to get to a position and actively read their positions, matching that up with whatever my requirements are for the swarm to do. That way, you aren't worried about actual program execution and only about what is happening in the simulation environment.

What I think is happening, and you mentioned this, is that .join() calls a program execution timer which waits for the action to finish and disrupts all of the other processes. In any of the swarm work I've done in AirSim, I don't touch the .join(), since it is, in my opinion, unrealistic to what actually occurs in real swarms.

Below is another algorithm I wrote to do a chase-follow algorithm. Please feel free to ask me questions about this as well. Here, I am checking to make sure that each drone is within a certain radius of the points that I want them to be in before giving out the next commands. Again, I never use .join(), except for when I am making sure that the drones are both up and ready to go. There is another library I haven't included that I wrote to generate paths and I can share that with you if you would like.

import setup_path 
import airsim

import numpy as np
import os
import tempfile
import pprint
import cv2
import time
from nav import Nav

veh_1_name = 'Lead'
veh_2_name = 'Follow'
waypoints_path_list = 'waypoints.json'
CONFIRMATION_DISTANCE = 1.5

# Load route
route = Nav(waypoints_path_list)
route.build_path()
print(len(route.primary_route))
print(route.primary_route[0])
print(route.primary_route[route.numb_waypoints - 1])

# connect to the AirSim simulator
client = airsim.MultirotorClient()
print(client)
client.confirmConnection()
print('Connection Confirmed')
client.enableApiControl(True, veh_1_name)
client.enableApiControl(True, veh_2_name)
client.armDisarm(True, veh_1_name)
client.armDisarm(True, veh_2_name)

airsim.wait_key('Press any key to takeoff')
f1 = client.takeoffAsync(vehicle_name=veh_1_name)
f2 = client.takeoffAsync(vehicle_name=veh_2_name)
f1.join()
f2.join()

state = client.getMultirotorState(vehicle_name=veh_1_name)
comms_data = client.getCommunicationsData(vehicle_name=veh_1_name)
print(veh_1_name, "\n")
print("State: %s" % pprint.pformat(state))
print("Comms Data: %s" % pprint.pformat(comms_data))

state = client.getMultirotorState(vehicle_name=veh_2_name)
print(veh_2_name, "\n")
print("State: %s" % pprint.pformat(state))

airsim.wait_key('Press any key to begin flying the route:')

# Fly the waypoint route. Ensure that the second vehicle triggers the join()
# so as the halt the loop.
for i, waypoint in enumerate(route.primary_route):
    try:
        next_point = route.primary_route[i + 1]
    except Exception as error:
        print('Final point!')
        next_point = False
    f1 = client.moveToPositionAsync(waypoint[0], waypoint[1], waypoint[2], waypoint[3], vehicle_name=veh_1_name)
    if (i % 10 == 0):
        f2 = client.moveToPositionAsync(waypoint[0], waypoint[1], waypoint[2] - 0.1, waypoint[3] + 1, vehicle_name=veh_2_name)
    else:
        f2 = client.moveToPositionAsync(waypoint[0], waypoint[1], waypoint[2], waypoint[3], vehicle_name=veh_2_name)

    if (i % 5 == 0):
        print(waypoint)

    # Instead of join, pull state and get GPS data to confirm where the drone is at
    kinematics = client.simGetGroundTruthKinematics(vehicle_name=veh_1_name)
    if next_point:
        # For the first point, the drones need to fly to a normalized location
        if i == 0:
            f2.join()
        else:
            while (abs(kinematics.position.x_val - abs(next_point[0])) - 3) > CONFIRMATION_DISTANCE and (abs(kinematics.position.y_val - abs(next_point[1])) - 3) > CONFIRMATION_DISTANCE and abs(abs(kinematics.position.z_val) - abs(next_point[2])) > CONFIRMATION_DISTANCE:
                kinematics = client.simGetGroundTruthKinematics(vehicle_name=veh_1_name)
                print('\n')
                print(kinematics.position.x_val)
                print(kinematics.position.y_val)
                print(kinematics.position.z_val)
                print((abs(kinematics.position.x_val - abs(next_point[0])) - 3))
                print((abs(kinematics.position.y_val - abs(next_point[0])) - 3))
                print(abs(abs(kinematics.position.z_val) - abs(next_point[2])))
                print('\n')
                time.sleep(0.2)

   # f1.join()

client.hoverAsync().join()

airsim.wait_key('Press any key to reset to original state')

client.armDisarm(False, veh_1_name)
client.armDisarm(False, veh_2_name)
client.reset()

# that's enough fun for now. let's quit cleanly
client.enableApiControl(False, veh_1_name)
client.enableApiControl(False, veh_2_name)

I also tried the first script with sleep times of 5 and 10 seconds to confirm that there isn't any ambiguity with calling the scheduling of drones to move independently of each other.

[GeneLYJ]

Thank you @xxEoD2242 for the 1st code. After having trying out and properly read your code, I understand why didn't my takeoff works for all my drones. To have the takeoff function and let the function run in a loop using .join() though I could see there's occasional sluggish during the loop but it's not as bad as using my code. You're mentioning about another library you haven't included, I assume is the nav library? It would be great if you can share with me. Thank you.

[xxEoD2242]

Hey @GeneLYJ , I can absolutely share that with you. It's really basic at the moment, where you just specify two points you want to move between and then it can break the route into multiple points. Myself and a team are working on broad scale swarm routing systems, which we will probably add as a feature to AirSim in the future.

Let me clean up the code and I'll throw it in a new repository and link that here. I'll add in the appropriate open-source licenses as well, so that you can use the code freely and not worry about any legal stuff.

[GeneLYJ]

Thanks a lot! That's great you and your team are going to release that feature. I'm still new to Python and still learning by myself. How are you able to come up something like that? I'm inspired.

[xxEoD2242]

Hey @GeneLYJ, here is the repository I promised. It's definitely a work in progress!

Multi Agent Routing

To answer your question, I currently do research with a controls professor from Purdue University and we use AirSim heavily. Together, we are working on building different swarm capabilities and our team hopes to add these capabilities to AirSim in the future. I'll post more about that later.

Feel free to modify or contribute to that repository as well. It's all open-source.

[GeneLYJ]

Thank you! I'll learn the concepts from your codes. That's cool to able to work together at the university. I'm still to AirSim and Python but I'll contribute whenever I see the possibilities.

[catproof]

you can also check to see if moveToPositionAsync is done executing by checking the result. example:

async_call = client.moveToPositionAsync(0, 1, 1, 1) if async_call.result != None: print("done moving to the position!")