PX4 HITL + External Offboard Controller + AirSim

[ibrhm0v]

Hi everyone and especially Mr. Chris Lovett (@lovettchris),

What is the best way to include my external offboard controller in my simulation environment?

I am trying to setup a HITL simulation environment for my drone using AirSim. I use a companion computer connected to autopilot through a serial port (a mavlink channel) that takes care of offboard control and some other stuff. I have been using AirSim since a long time and am quite familiar with the sources. I know that AirSim is ready for PX4 HITL and that its configuration is straight forward through settings.json (UseSerial and etc.). Here are my questions:

Question 1: What I have tried to do first was just set AirSim as HITL in settings.json as described in the docs to connect my autopilot to the simulator through USB (ttyACMx) and connect my offboard controller to my autopilot through a serial port as it is in the real system. This caused some instabilities in flights, actually sometimes it led to a meaningless freefall after some flight time (3-5 minutes). I actually couldn't find a reason for this either in airSim or px4 logs. But I think airSim is somehow and somewhat interfering with offboard controller mavlink messages. Can we discuss this? What do you think is wrong with my setup?

Question 2: After failing my first solution, I thought of connecting the autopilot to my offboard controller through airSim and implemented an offboard node in MavLinkMultirotorApi.hpp and have tried to "route" offboard controller messages to my pixhawk and pixhawk messages to offboard controller. But I think I am misunderstanding the code. Do you think I am on the right path?

Thanks

Ubuntu 20.04 LTS AirSim v1.7.0 (Built from minorly customized source) UE4.25

`{

"SettingsVersion": 1.2,
"SimMode": "Multirotor",
"Wind": {
    "X": 0,
    "Y": 0,
    "Z": 0
},
"PawnPaths": {
    "DefaultQuadrotor": {
        "PawnBP": "Class'/Game/Blueprints/BP_FlyingPawn.BP_FlyingPawn_C'"
    }
},
"DefaultSensors": {
    "Barometer": {
        "SensorType": 1,
        "Enabled": true
    },
    "Imu": {
        "SensorType": 2,
        "Enabled": true
    },
    "Gps": {
        "SensorType": 3,
        "Enabled": true
    },
    "Magnetometer": {
        "SensorType": 4,
        "Enabled": true
    },
    "Lidar": {
        "SensorType": 6,
        "Enabled": false,
        "NumberOfChannels": 4,
        "PointsPerSecond": 10000
    }
},
"OriginGeopoint": {
    "Latitude": 39.9334,
    "Longitude": 32.8597,
    "Altitude": 889
},
"Vehicles": {
    "PX4": {
        "VehicleType": "PX4Multirotor",
        "Model": "MyCustomMultirotor",
        "SerialBaudRate": 921600,
        "SerialPort": "*",
        "UseSerial": true,
        "UseTcp": false,
        "ControlIP": "127.0.0.1",
        "ControlPortLocal": 14580,
        "ControlPortRemote": 14540,
        "UdpIp": "",
        "UdpPort": 14540,
        "TcpPort": 4560,
        "VehicleCompID": 1,
        "VehicleSysID": 142,
        "LocalHostIp": "127.0.0.1",
        "params": {
            "NAV_RCL_ACT": 0,
            "NAV_DLL_ACT": 0,
            "LPE_LAT": 47.641468,
            "LPE_LON": -122.140165,
            "COM_OBL_ACT": 1
        }
    }
}

}`

[lovettchris]

I have a couple ideas: 1) Have you tried it without the Lidar sensor - the Lidar is expensive in CPU and this could be causing AirSim to run too slowly, and send messages to the Pixhawk too slowly causing the simulation to fall behind. 2) But falling behind is solved by running PX4 in "lockstep" mode. See Lockstep Mode and follow those steps to ensure the pixhawk is in lockstep mode then the simulation should work well even when it is running slowly.

[ibrhm0v]

@lovettchris thanks, I'll give lockstep a try and will remove lidar as I do not need it anymore. Will update here

[ibrhm0v]

Lockstep is for tcp connection i.e SITL right? I am trying to achieve this with a serial connection

[ibrhm0v]

I removed the Lidar sensor from settings but still failed (with lockstep enabled and disabled) to do a stable flight. In order to concentrate my focus I thought of removing all my custom components from the system. So that, I downloaded the pre-built AirSimNH binaries (v1.7.0), uploaded a px4 v1.12.3 (again pre-built) binary on a Pixhawk 2 Cube, removed the offboard controller and replaced my custom ground control station with QGC. So the system is now composed of these default binaries and qgc. I connected my flight controller through USB (ttyACM0 @ 921600). Again the same problem occurs. Drone suddenly starts falling.

When I checked the flight log file (ulg). It seems like simulator fails to send sensor data after a certain time, which is the same moment when the fall starts. Here is the flight log: log_0_2022-5-17-17-53-12.zip.

Here is a video of what happens: https://user-images.githubusercontent.com/42241276/168849227-d73ff350-92f3-439e-9ea2-6faf6febcdd9.mp4

By the way I think my system must be capable of running AirSim. Here are the specs FYI:

• 12th Gen Intel(R) Core(TM) i7-12700KF
• Nvidia Geforce RTX3080 Ti 12Gb
  • 64Gb RAM
  • Ubuntu 20.04 LTS
  • Everything running on SSD

[ibrhm0v]

By the way this happens after like 3.00-3.15 minutes of healthy flight. The video is shortened for size

[lovettchris]

Thanks for the video. Yes, your system is more than capable of running AirSim. It does appear like lockstep got stuck. It has the classic symptoms. I'm recalling that a normal pixhawk firmware build (designed for flying real drones) does NOT have lockstep enabled by default which means you will have to build the firmware with lockstep enabled. But can you try another test before you do that. Can you try and disable lockstep by explicitly setting it to false in the AirSim settings.json file? Lockstep will only work if both sides have it enabled. If you tell AirSim not to use lockstep then it won't get in a "stuck state" waiting for a lockstep reply that is never coming.

[ibrhm0v]

I gave it a shot. Unfortunately the result was same. I'll give it another try tomorrow

[ibrhm0v]

What I don't understand is; ain't lockstep an SITL feature? Can it be activated in HITL also? (https://docs.px4.io/v1.12/en/simulation/#lockstep-simulation)

[lovettchris]

Right, you can see in the code that it is only enabled for SITL, here:

You are using HITL, and so I'm wondering if you could build the HITL firmware with lockstep enabled, things might be a lot more predictable and you can run the simulation as slow as you want (e.g. with Lidar and anything else you need) even step through the code in the debugger and everything just works.

I'm surprised it gets stuck when lockstep is disabled. Can you confirm you set the lockstep to false with "LockStep": false,", rather than removing lockstep setting? Note: if you go down the custom HITL with lockstep enabled road you will also need to switch to the SteppableClock.

[ibrhm0v]

I'll try these on Monday and will update you. Thanks for keeping up

[ibrhm0v]

I tried building PX4 (HITL) with lockstep enabled but what I have found out that the lockstep_scheduler implementation is only available for posix. Therefore it has to be ported for nuttX. So, currently, there is no implementation of lockstep in PX4 except for SITL.

[ibrhm0v]

Setting lockstep to "false" in settings.json doesn't help. After 3 minutes of normal flight. Same thing happens :/

[lovettchris]

Ok, thanks for trying, one last thing to try, can you try and move your initial position to GPS coordinates for Seattle. There was a long time ago a bug in AirSim related to compass sensor that resulted in slight crabbing behavior of the drone if you flew elsewhere in the world (I think the computation of true north disagrees with the algorithm in PX4 and so they are off a bit). I wonder if this could accumulate over a long flight to create bigger problems...

[ibrhm0v]

I have set the OriginGeopoint lat and lons and px4 LPE_LAT/LON to Redmond. Same thing happens... I wonder if I can trace anything through AirSim logs? When I check the contents of "Saved" folder, everything seems to be normal. Only that px4 logs say ACCEL #0 TIMEOUT and BARO #0 TIMEOUT.

[ibrhm0v]

@Jaeyoung-Lim any ideas about this?

[ibrhm0v]

Hi, did anyone had the chance to reach any conclusion with this?

[ibrhm0v]

@Jaeyoung-Lim @lovettchris is there any news about this?