IMU through MAVROS

[plushpluto]

I am trying to benchmark VIO pipelines (MSCKF, OKVIS, ROVIO, VINS-Mono, VINS-Fusion, SVO+MSF, SVO+GTSAM, and CNN-SVO) using AirSim.

While in Gazebo/EuRoC MAV Dataset, it works. But there are some discrepancies while taking the IMU values from AirSim.

What is the correct way to obtain the IMU values through MAVROS? Please advise!

[rajat2004]

I'm not completely sure since haven't played around much with the ROS wrapper, but you'll probably have to do some conversion to convert messages from sensor_msgs::Imu to the required Mavros format. I think the ROS wrapper publishes the messages to /airsim_node/imu/<sensor_name> - https://github.com/microsoft/AirSim/blob/master/ros/src/airsim_ros_pkgs/src/airsim_ros_wrapper.cpp#L210 You might also want to have a look at https://github.com/microsoft/AirSim/pull/2743 which does quite a few additions and improvements to the wrapper

[catproof]

@plushpluto @rajat2004 did you ever get this to work? I am having some issues setting up the IMU parameters for testing SLAM algorithms, see my issue: https://github.com/microsoft/AirSim/issues/4119

[xxEoD2242]

Make sure to define a sensor in your settings.json but label it as:

"imu"

and not Imu.

The labels are incorrect in the documentation.

Then you access it like @rajat2004 said. Had good luck with this but the IMU is rather noisy and there some open issues about whether the simulated IMU is correct or not.

[catproof]

i'm not sure what you mean whether the IMU is correct or not. I in this thread: https://github.com/microsoft/AirSim/issues/2369 someone got VINS-Fusion to work with AirSim

"For problem 234, my modification loses the generic but it solves the problem, gives me 20HZ stereo images and 300hz IMU on a remote notebook and successfully running VINS-Fusion in AirSim smoothly, here is my repo"

it looks like the noise parameters they use are somewhat arbitrary and do not follow the same parameters determined by this user (which they determined both analytically and experimentally).

I am still a bit uncertain as to how to set up my camera-to-imu transformation matrix. (i.e. determining the orientation of the camera compared to the IMU). There is nothing in the documentation saying what the default orientation of the IMU sensor is.

[xxEoD2242]

@NickPerezCarletonUniversity

We used the same parameters and noticed that there wasn't any difference when we actually went through the AirSim code and calculated the correct values for those parameters in VINS-Fusion.

Here is the camera-to-imu transformation we got to work. The reason that this works is that the camera is in Unreal Engine coordinate frame of and the imu is NED coordinate frame.

extrinsicRotation: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [0,0,1,
          1,0,0,
          0,1,0]

Other then that, we had these parameters::

acc_n: 0.1 #0.00273735          # accelerometer measurement noise standard deviation. #0.2   0.04
gyr_n: 0.01 #0.00058877         # gyroscope measurement noise standard deviation.     #0.05  0.004
acc_w: 0.0001 #0.00015193        # accelerometer bias random work noise standard deviation.  #0.02
gyr_w: 0.00001 #7.878e-5       # gyroscope bias random work noise standard deviation.     #4.0e-5
g_norm: 9.81007     # gravity magnitude

The commented out ones are the ones we calculated using the IMU code.

[xxEoD2242]

If you run the automated extrinsic rotation calculation through VINS-Fusion, you will get nearly the same rotation matrix btw

[catproof]

We used the same parameters and noticed that there wasn't any difference when we actually went through the AirSim code and calculated the correct values for those parameters in VINS-Fusion.

sorry... I am a bit confused by this. are you saying that there was no difference between the noise parameters you calculated based off the AirSim code and the ones you calculated using VINS-Fusion?... why did you comment out the parameters you derived from the IMU code?

[catproof]

how did you get the parameters?: acc_n: 0.1 gyr_n: 0.01 acc_w: 0.0001 gyr_w: 0.00001

[catproof]

Here is the camera-to-imu transformation we got to work.

did you change the position of your camera at all? or are you using the default position for the camera? I noticed you are only doing a rotation, and no translation. Your rotation matrix agrees with the one in this config file, it looks like the difference in sign is because you are doing camera-to-imu, and they are doing imu-to-camera.

[catproof]

@xxEoD2242 you can also use something like Kalibr: https://github.com/ethz-asl/kalibr to calculate the noise parameters for the IMU from AirSim. that is how this person in this issue: https://github.com/microsoft/AirSim/discussions/3523#discussioncomment-525240 calculated the parameters experimentally.

[catproof]

also, with regards to this issue: https://github.com/microsoft/AirSim/issues/4133 do you know whether or not the getImuData() method in AirSim uses degrees or radians?...

[xxEoD2242]

also, with regards to this issue: #4133 do you know whether or not the getImuData() method in AirSim uses degrees or radians?...

It's in radians

[xxEoD2242]

Here is the camera-to-imu transformation we got to work.

did you change the position of your camera at all? or are you using the default position for the camera? I noticed you are only doing a rotation, and no translation. Your rotation matrix agrees with the one in this config file, it looks like the difference in sign is because you are doing camera-to-imu, and they are doing imu-to-camera.

Yeah, we had some x-axis translation that we added as well. That doesn't affect the rotation matrix however

[xxEoD2242]

how did you get the parameters?: acc_n: 0.1 gyr_n: 0.01 acc_w: 0.0001 gyr_w: 0.00001

Those are the default ones used in the EUROC dataset. They wound up working better then the calculated values.

Your core problem with VINS-Fusion and VINS-Mono stems from the fact that the feature matching system of this VIO strategy just doesn't work well in the simulated environments. If you watch the outputs of feature matching, you will see that the system does identify the points sometimes, but it struggles with most scenery. Potentially, with a more complex environment, you would get better results.

Another key struggle is how much the IMU fluctuates, again because the simulated data isn't the best. We confirmed all of this experimentally in multiple environments with multiple parameter settings. We used both the default parameters and the calculated ones and saw no difference in performance.

[xxEoD2242]

@xxEoD2242 you can also use something like Kalibr: https://github.com/ethz-asl/kalibr to calculate the noise parameters for the IMU from AirSim. that is how this person in this issue: #3523 (comment) calculated the parameters experimentally.

Yeah, we tried this. Parameters didn't work any better then what was listed before. Issue is the feature matching and not the IMU

[catproof]

Yeah, we had some x-axis translation that we added as well. That doesn't affect the rotation matrix however

I see. By default the camera and the IMU are both located at the same position though, right? So if you translated the camera position by [x',y',z'], you would incorporate a translation of [-x',-y',-z'] in the camera-to-imu transformation, correct?

[catproof]

Another key struggle is how much the IMU fluctuates, again because the simulated data isn't the best. We confirmed all of this experimentally in multiple environments with multiple parameter settings. We used both the default parameters and the calculated ones and saw no difference in performance.

As a (somewhat) quick test, you could try running your algorithm on the TartanAir dataset. It was generated using AirSim and has very high quality environments. If you still have issues on that dataset, I would imagine you won't get better results in other environments (maybe that will change if/when AirSim upgrades to use some of the incredible visuals in Unreal Engine 5). I noticed the features used in Orb Slam (Orb features) struggle quite a bit with the environment I'm using (which is quite realistic looking). I got over this issue by slowing down the speed of the drone. If the drone moves too fast it looses tracking consistently.

[catproof]

Your rotation matrix agrees with the one in this config file, it looks like the difference in sign is because you are doing camera-to-imu, and they are doing imu-to-camera.

I just realized you rotation matrix does not agree with what this user has in their config file. If you take the inverse of their imu-to-camera matrix (to get their camera-to-imu matrix), they get a resulting rotation of [0, -1, 0 0, 0, -1 1, 0, 0]

[xxEoD2242]

@NickPerezCarletonUniversity

Yup, that's what we saw as well. That's why we didn't use theres.

The Tartan dataset is great but there is an issue. It's the timing stamp of the IMU matching with the features that is the problem. In real-time simulation, this doesn't work, as the timestamps differ wildly, as can be seen by estimating TD with VINS Mono or Fusion.