Jittery video capture with repeated moveByVelocityAsync()

[nethtrick]

Hello, I am trying to record smooth videos of a drone camera looking in a fixed direction while repeatedly calling moveByVelocityAsync() to follow a path. The captured images show a lot of jitter, even though I am in querying and in theory compensating for the drone's orientation. There is no jitter when the velocity is constant, but by calling moveByVelocityAsync() repeatedly with slightly different velocities, the queried orientation seems to not be in sync with the images.

I have included a simplified Python script below to reproduce the effect along with an example settings.json. I am using Anaconda Python 3.7 with AirSim 1.3.1.

I am trying to capture images at a fixed framerate (simulation time). So I have a main loop which performs the following at each iteration:

1. Call moveByVelocityAsync() with velocities slightly perturbed around a mean velocity
2. Call simContinueForTime() which propagates one frame duration and then pauses the game.
3. Call getGroundTruthKinematics() to query the current orientation of the drone
   • It seems like this is not returning the correct orientation
4. Transform the desired camera direction in world coordinates to yaw, pitch, roll in the drone's body frame
5. Call simSetCameraOrientation() to set the camera's orientation
   • Ideally, this would result in a fixed orientation in world coordinates, but seems to be jittery
6. Capture images at the current (paused) game time

The script below shows what happens when random perturbations are added to the mean velocity, but in theory there would be no jitter even with these perturbations since getGroundTruthKinematics() is being called to query the exact orientation at each frame. Also, calling time.sleep(1) does not seem to help between simContinueForTime() and getGroundTruthKinematics().

Please let me know if you have any ideas what is happening or how to fix this. I even bumped up the priority of the rendering thread but to no avail.

Note: this seems to be worse for higher resolution images so maybe there is something changing during the rendering time, even though the game should be paused. I have also tried on a very capable desktop computer (4x Intel Xeon & Nvidia RTX 2080 Ti) but the problem remains.

Python script:

# Python std libs
import os, sys, argparse, numpy, time, tqdm
import scipy
import scipy.spatial
# AirSim
import airsim

# Main program
if __name__ == "__main__":

    # Folder for captured images
    folder = 'images'
    if not os.path.isdir(folder):
        os.makedirs(folder)

    # Camera name (as in settings.json)
    cam = '0'

    # Initialize AirSim
    cl = airsim.MultirotorClient()
    cl.confirmConnection()
    # Establish control
    cl.enableApiControl(True)

    # Takeoff    
    cl.takeoffAsync().join()

    # Flight time
    fly_time = 5.0
    # Horizontal speed
    speed = 5.0
    # Average velocity
    vel = [5,5,-3]

    # Desired camera direction (inertial coords)
    r0 = [0,1,0.5]

    # Fly drones
    def fly_drone(vel_spread=0.):
        # Cancel last task
        cl.cancelLastTask()
        # move by randomized velocity
        v = numpy.random.normal(vel,vel_spread)
        cl.moveByVelocityAsync(*v,duration=fly_time)

    # Initial move command
    fly_drone()

    # Timing vars
    period = 1./60.
    steps = int(round(fly_time/period))

    # Image requests
    requests = [airsim.ImageRequest(cam, airsim.ImageType.Scene,
                                      False,True)]

    # Image responses
    responses = []

    # Main loop
    for i in tqdm.tqdm(range(steps)):

        # Fly
        # No jitter:
        #fly_drone(0)
        # Produces jitter:
        fly_drone(0.5)

        # Continue for one frame
        cl.simContinueForTime(period)

        # Sleeping doesn't seem to help
        #time.sleep(1)

        # Get ground truth kinematics
        kin = cl.simGetGroundTruthKinematics()

        # Get drone 0 orientation
        orient = kin.orientation.to_numpy_array()
        q = scipy.spatial.transform.Rotation.from_quat(orient) 

        # Get drone 1 position in body frame of drone 0
        r = q.inv().apply(r0)

        # Get yaw, pitch, roll
        ypr = [numpy.arctan2(r[1],r[0]),
               -numpy.arcsin(r[2]/numpy.linalg.norm(r)),
               0.0]

        # Camera orientation in drone 0 body frame, as quaternion
        q = scipy.spatial.transform.Rotation.from_euler('ZYX',ypr).as_quat()

        # Set camera orientation
        cl.simSetCameraOrientation(cam, airsim.Quaternionr(*q) ) 

        # Now, capture images        
        responses.append( cl.simGetImages(requests) )

    # Save image data
    for i in range(len(responses)):
        # Generate a filename
        filename = os.path.join(folder,'{:05d}.png'.format(i))
        # The image data
        data = responses[i][0].image_data_uint8
        # Write image data
        with open(filename,'wb') as FILE:
            # Write the bytes to file
            FILE.write(data)

    # Reset sim
    cl.reset()

settings.json:

{
    "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
    "SettingsVersion": 1.2,
    "SimMode": "Multirotor",
    "Vehicles": 
    {
        "Drone0": 
        {
            "Cameras": 
            {
                "0": 
                {
                    "CaptureSettings": 
                    [
                        {
                            "ImageType": 0,
                            "Width": 1920,
                            "Height": 1080,
                            "FOV_Degrees": 90,
                            "MotionBlurAmount": 0,
                            "TargetGamma": 1.0,
                            "ProjectionMode": ""
                        }
                    ],
                    "Gimbal": 
                    {
                        "Stabilization": 0,
                        "Pitch": 0,
                        "Roll": 0,
                        "Yaw": 0
                    },
                    "X": 0,
                    "Y": 0,
                    "Z": 0.4,
                    "Pitch": 0,
                    "Roll": 0,
                    "Yaw": 0
                }   
            },
            "VehicleType": "SimpleFlight",
            "DefaultVehicleState": "Armed",
            "IsFpvVehicle": true,
            "ViewMode":  "FPV"
        }
    }
}

[ACLeighner]

Check to see if your drone is physically shaking at higher altitude (>60) It is a known bug with the PID controller. I posted a fix for it here: https://github.com/microsoft/AirSim/issues/1394#issuecomment-629936379

[nethtrick]

Hello ACLeighner,

Thank you for the post. I looked at your problem and glad you got it fixed. I will test your fix to see if that makes a difference and provide an update, it's possible. I'm not sure it would explain the camera jitter I'm seeing but maybe. I had noticed problems flying at higher altitudes in the Blocks environment and had ascribed it to some sort of "ceiling" on that particular level, and achieved flight at higher altitudes on another (custom) level. Maybe try that as well.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity from the community in the last year. It will be closed if no further activity occurs within 20 days.

[azadhamid]

Hello, Could you solve the problem? I have the same problem and want to have a "smooth flying" and "recording" while calling moveByVelocityAsync several times.

[xxEoD2242]

@azadhamid We resolved this issue awhile ago by writing our own Position Controller and increasing the update rate to 50 Hz.

Also, AirSim has been shutdown and will be archived soon. We continued the open source project here: Colosseum.

If you are looking for more advanced features including automated data collection, you can checkout our platform: SWARM Developer Platform, which gives you access to our custom controllers, automated data collection and more.

Here is an example: SWARM Developer - House Flythrough

[azadhamid]

@azadhamid We resolved this issue awhile ago by writing our own Position Controller and increasing the update rate to 50 Hz.

Also, AirSim has been shutdown and will be archived soon. We continued the open source project here: Colosseum.

If you are looking for more advanced features including automated data collection, you can checkout our platform: SWARM Developer Platform, which gives you access to our custom controllers, automated data collection and more.

Here is an example: SWARM Developer - House Flythrough

Thank you for your suggestion. Interesting, I will check the links. Can I have access to the "Position Controller" you mentioned? Thanks