search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.23k stars 13.39k forks source link

Consistently bad magnetometer calibration on 1.12 #18041

Closed le-chat closed 3 years ago

le-chat commented 3 years ago

We repeatedly get bad magnetometer calibration results on 1.12: after aircraft rotation yaw drifts. See https://logs.px4.io/plot_app?log=13194004-cc98-49e9-80d2-e1e1aa7196f7 for example. On magnetometer raw data and gyro data plots there are very clear time ranges, when actual rotation happens, but yaw plot demonstrates drift outside of them. Also yaw check bit is 1 in estimator watchdog.

It isn't a hardware issue, because when we switch back to 1.13 firmware we get good results: https://logs.px4.io/plot_app?log=d4f79fd4-1623-43ad-baa6-bd748f4a1e2e.

We have also recorded log on 1.12 with SENS_BOARD_ROT=0 to check if this relates to problem, still getting drift: https://logs.px4.io/plot_app?log=3760a163-3aa2-48f0-84f5-afdacd6dd3b4.

mengchaoheng commented 3 years ago

the same probrem, Consistently always troubles me

dagar commented 3 years ago

I see the build is px4_fmu-v3. Exactly which flight controller are you using?

dagar commented 3 years ago

Just to clarify against the logs can you share more details about the physical setup? How is the flight controller mounted, how are the external magnetometers mounted?

dagar commented 3 years ago

For reference mag calibration auto rotation (CAL_MAG_ROT_AUTO) only works if the board has an internal magnetometer that is mounted correctly. The rotations of the external sensors are updated to align with the onboard mag.

What's a bit confusing in your log is the first mag (internal?) is CAL_MAG0_ROT 24.

le-chat commented 3 years ago

Pixhawk 2.4.8 from https://a.aliexpress.com/_uxUElW This is tailsitter duo, pixhawk is mounted with arrow up. Log for 1.11.3 contains right orientation for both flight controller and compass.

(The log with SENS_BOARD_ROT=0 was written and calibrated with aircraft in fixed-wing position, two other logs in normal position as for MC mode.)

le-chat commented 3 years ago

Yes, mag0 is internal magnetometer, CAL_MAG0_ROT always equals SENS_BOARD_ROT.

rizvanovic commented 3 years ago

We've also had a lot of calibration and mag issues since upgrading to 1.12, but did a considerable HW change in the process. In our case it could very well be EMI from a companion computer or radio setup.

le-chat commented 3 years ago

We have tested 1.11 and 1.12 on the exactly same hardware, all three logs above were written in one place in one time, no modification of aircraft was done.

dagar commented 3 years ago

We've also had a lot of calibration and mag issues since upgrading to 1.12, but did a considerable HW change in the process. In our case it could very well be EMI from a companion computer or radio setup.

Please share any logs (+ context) you have n hand that might show an issue.

dagar commented 3 years ago

@le-chat

Parameters in the first log (https://logs.px4.io/plot_app?log=13194004-cc98-49e9-80d2-e1e1aa7196f7)

CAL_MAG0_ID, 396817
CAL_MAG0_PRIO, 0
CAL_MAG0_ROT, 24  # ROTATION_PITCH_90

CAL_MAG1_ID, 527625
CAL_MAG1_PRIO, 75
CAL_MAG1_ROT, 20  # ROTATION_ROLL_270

SENS_BOARD_ROT, 24 # ROTATION_PITCH_90

Mag 0 is the internal lsm303d. Did you set CAL_MAG0_ROT manually? These rotation parameters should actually be set to -1 if they're probably registered as internal, so I'm trying to figure out what's happening here.

Mag 1 is an AK09916 (Here module?). How is it mounted on the vehicle?

le-chat commented 3 years ago

Yes, CAL_MAG0_ROT has been set manually during our experiments.

Magnetometer is https://a.aliexpress.com/_vaN7Di, it's connected via I2C (together with airspeed sensor), is mounted on tail.

dagar commented 3 years ago

Ok, I think it's an interaction with mag 0 being disabled (CAL_MAG0_PRIO 0) and then not properly being updated (internal/external), then at calibration time it's not being used correctly to set the rotation of the external.

Additionally the rotation of the AK09916 driver changed to actually respect the sensor and not the arbitrary mounting in the Here GPS. That's why the CAL_MAG1_ROT from v1.11 is wrong in v1.12, and this is also one of the reasons why we needed to introduce mag auto rotation in the first place.

@le-chat with v1.12 could you try re-enabling Mag 0 (eg CAL_MAG0_PRIO 50), then do a fresh magnetometer calibration? On success CAL_MAG0_ROT should be -1 (internal mag) and CAL_MAG1_ROT should be something else other than 20 (I need to double check the actual mounting).

If this is the problem then I think I can make a quick fix for v1.12.2 that prevents people from falling into this trap.

dagar commented 3 years ago

Working on a fix in https://github.com/PX4/PX4-Autopilot/pull/18046.

dagar commented 3 years ago

Magnetometer is https://a.aliexpress.com/_vaN7Di, it's connected via I2C (together with airspeed sensor), is mounted on tail.

@le-chat that link says HMC5883L, but your log shows AK09916. Does that make sense or is there a mistake somewhere?

dagar commented 3 years ago

@mengchaoheng any details you can share? Vehicle set up, logs, etc?

le-chat commented 3 years ago

Hmm, I know only that this magnetometer did work on 1.11.3... Attaching photos magnetometer-1 magnetometer-2

By the way, we cannot set internal orientation for MAG0, it automatically resets to 0 both via interface and console:

nsh> param set CAL_MAG0_ROT -1
+ CAL_MAG0_ROT: curr: 0 -> new: -1
nsh> param show CAL_MAG0_ROT
Symbols: x = used, + = saved, * = unsaved
x + CAL_MAG0_ROT [67,129] : 0

 907/1544 parameters used.
nsh>

Will try to calibrate with CAL_MAG0_PRIO=50 and CAL_MAG0_ROT=SENS_BOARD_ROT soon.

le-chat commented 3 years ago

After calibration with priority 50 yaw again drifts: https://logs.px4.io/plot_app?log=4bc3697d-4773-4bbb-aaf1-d458042ea0cb. CAL_MAG1_ROT=0, CAL_MAG0_ROT=24.

Log with calibration procedure itself, if this may be of interest: https://logs.px4.io/plot_app?log=41200dab-3f6b-48bc-a050-90745df7bdcb.

dagar commented 3 years ago

On your board with v1.12 can you check a few things?


nsh> listener sensor_mag

nsh> sensors status

nsh> listener vehicle_magnetometer

nsh> work_queue status
RescueBeaver commented 3 years ago

Hi, I`m pilot from the same team.

nsh> listener sensor_mag

TOPIC: sensor_mag 2 instances

Instance 0:
 sensor_mag_s
    timestamp: 276668773  (0.013218 seconds ago)
    timestamp_sample: 276667757  (1016 us before timestamp)
    device_id: 396817 (Type: 0x06, I2C:2 (0x0E)) 
    x: 0.1560
    y: -0.3901
    z: -0.0151
    temperature: nan
    error_count: 0
    is_external: True

Instance 1:
 sensor_mag_s
    timestamp: 276711863  (0.001902 seconds ago)
    timestamp_sample: 276710907  (956 us before timestamp)
    device_id: 527625 (Type: 0x08, I2C:1 (0x0D)) 
    x: -0.4060
    y: -0.2462
    z: 0.1254
    temperature: nan
    error_count: 0
    is_external: True
nsh> sensors status
INFO  [sensors] selected gyro: 2162698 (0)
INFO  [data_validator] validator: best: 0, prev best: 0, failsafe: NO (0 events)
INFO  [data_validator] sensor #0, prio: 50, state: OK
INFO  [data_validator]     val:   0.0026, lp:   0.0033 mean dev:   0.0000 RMS:   0.0007 conf:   1.0000
INFO  [data_validator]     val:  -0.0008, lp:  -0.0003 mean dev:   0.0000 RMS:   0.0007 conf:   1.0000
INFO  [data_validator]     val:  -0.0005, lp:  -0.0003 mean dev:   0.0000 RMS:   0.0008 conf:   1.0000

INFO  [sensors] selected accel: 2162698 (0)
INFO  [data_validator] validator: best: 0, prev best: 0, failsafe: NO (0 events)
INFO  [data_validator] sensor #0, prio: 50, state: OK
INFO  [data_validator]     val:   1.3151, lp:   1.2884 mean dev:  -0.0002 RMS:   0.0252 conf:   1.0000
INFO  [data_validator]     val:  -0.7150, lp:  -0.7343 mean dev:   0.0001 RMS:   0.0161 conf:   1.0000
INFO  [data_validator]     val:  -9.7938, lp:  -9.7647 mean dev:   0.0001 RMS:   0.0159 conf:   1.0000

INFO  [vehicle_magnetometer] selected magnetometer: 396817 (0)
INFO  [data_validator] validator: best: 0, prev best: 0, failsafe: NO (0 events)
INFO  [data_validator] sensor #0, prio: 50, state: OK
INFO  [data_validator]     val:   0.2754, lp:   0.2772 mean dev:  -0.0000 RMS:   0.0038 conf:   1.0000
INFO  [data_validator]     val:  -0.1068, lp:  -0.1082 mean dev:  -0.0000 RMS:   0.0026 conf:   1.0000
INFO  [data_validator]     val:  -0.2659, lp:  -0.2678 mean dev:   0.0000 RMS:   0.0017 conf:   1.0000
INFO  [data_validator] sensor #1, prio: 50, state: OK
INFO  [data_validator]     val:  -0.3467, lp:  -0.3493 mean dev:   0.0000 RMS:   0.0022 conf:   1.0000
INFO  [data_validator]     val:  -0.3625, lp:  -0.3606 mean dev:  -0.0000 RMS:   0.0021 conf:   1.0000
INFO  [data_validator]     val:   0.1543, lp:   0.1535 mean dev:  -0.0000 RMS:   0.0021 conf:   1.0000
INFO  [sensor_calibration] MAG 396817 EN: 1, offset: [-0.060 -0.258 -0.277], scale: [ 1.085  0.884  1.057], External ROT: 24
INFO  [sensor_calibration] MAG 527625 EN: 1, offset: [-0.049  0.114 -0.028], scale: [ 0.986  0.991  1.032], External RO
nsh> listener vehicle_magnetometer

TOPIC: vehicle_magnetometer
 vehicle_magnetometer_s
    timestamp: 320882436  (0.005593 seconds ago)
    timestamp_sample: 320881346  (1090 us before timestamp)
    device_id: 396817 (Type: 0x06, I2C:2 (0x0E)) 
    magnetometer_ga: [0.2811, -0.1066, -0.2681]
    calibration_count: 3
nsh> work_queue status

Work Queue: 8 threads                        RATE        INTERVAL
|__ 1) wq:rate_ctrl    
|   |__ 1) mc_rate_control               399.7 Hz         2502 us
|   |__ 2) vehicle_angular_velocity      399.7 Hz         2502 us
|   \__ 3) vtol_att_control              597.5 Hz         1674 us
|__ 2) wq:SPI1         
|   |__ 1) mpu6000                       399.5 Hz         2503 us
|   \__ 2) ms5611                         99.3 Hz        10072 us
|__ 3) wq:I2C1         
|   |__ 1) ms4525_airspeed                92.9 Hz        10770 us
|   \__ 2) qmc5883l                       50.0 Hz        20004 us (20000 us)
|__ 4) wq:I2C2         
|   |__ 1) ist8310                        46.3 Hz        21588 us
|   \__ 2) rgbled                         38.5 Hz        25964 us
|__ 5) wq:nav_and_controllers
|   |__ 1) airspeed_selector              10.0 Hz        99985 us (100000 us)
|   |__ 2) flight_mode_manager            50.0 Hz        19998 us
|   |__ 3) fw_att_control                199.9 Hz         5004 us
|   |__ 4) fw_pos_control_l1              50.0 Hz        19998 us
|   |__ 5) land_detector                  99.7 Hz        10034 us
|   |__ 6) mc_att_control                199.9 Hz         5004 us
|   |__ 7) mc_hover_thrust_estimator      99.9 Hz        10007 us
|   |__ 8) mc_pos_control                 99.6 Hz        10039 us
|   |__ 9) sensors                       199.6 Hz         5011 us
|   |__10) vehicle_acceleration          200.4 Hz         4990 us
|   |__11) vehicle_air_data               74.5 Hz        13429 us
|   |__12) vehicle_gps_position            5.0 Hz       199737 us
|   \__13) vehicle_magnetometer           92.6 Hz        10794 us
|__ 6) wq:INS0         
|   |__ 1) ekf2                          199.4 Hz         5014 us
|   \__ 2) vehicle_imu                   200.1 Hz         4996 us
|__ 7) wq:hp_default   
|   |__ 1) battery_status                100.0 Hz        10000 us
|   |__ 2) board_adc                     100.0 Hz        10000 us (10000 us)
|   |__ 3) rc_update                       0.0 Hz            0 us
|   \__ 4) tone_alarm                      0.1 Hz     17116558 u
dagar commented 3 years ago

Thanks @RescueBeaver, I see what's going on now.

Ignore what I said earlier about the specific magnetometers, I incorrectly decoded the device ids.

MAG0 - IST8310 on I2C2, device_id: 396817 (Type: 0x06, I2C:2 (0x0E)) MAG1 - QMC5883L on I2C1, device_id: 527625 (Type: 0x08, I2C:1 (0x0D))

Where is the IST8310? It definitely wasn't part of the original pixhawk (or any fmu v2/v3), but is it now being used as the only internal mag? If you can verify the specific orientation relative to the pixhawk direction of flight I can show you how to get it started properly. Once the internal magnetometer is correct the mag calibration in v1.12 will automatically set the rotation of the external (qmc5883l) properly.

RescueBeaver commented 3 years ago

The problem is that it is internal. May be we can verify its orientation from logs for 1.11.3?

dagar commented 3 years ago

Actually, maybe this is easier.

Exactly how is the qmc5883l mounted in the vehicle body frame? We can simply set its rotation correctly manually and be done.

RescueBeaver commented 3 years ago

It was working correctly with 1.11.3 with rotation roll 270.

dagar commented 3 years ago

The compass rotations in the drivers were a complete mess with different manufacturers mounting them differently on compass/GPS units. https://github.com/PX4/PX4-Autopilot/pull/15120

Since then they've all been normalized to be consistent (https://github.com/PX4/PX4-Autopilot/pull/14384), but that means some are wrong in older parameter sets. This was acceptable because we added auto rotation during calibration, but since your pixhawk board doesn't technically have an internal mag that isn't working properly either.

With the drivers untangled it's now much easier to just figure out the magnetometer rotation directly from how it's physically mounted instead of playing the guessing game. You can also trace through and figure out how things changed in the driver, but that can quickly get confusing.

Can you try CAL_MAG1_ROT 18 (ROTATION_ROLL_90_YAW_90)?

RescueBeaver commented 3 years ago

Ive tried, it doesnt work.

RescueBeaver commented 3 years ago

I can watch for mags outputs by listener sensor_mag. Can it help to verify mags orientation?

dagar commented 3 years ago

Typically I use the internal mag to set things, but in this case both are probably wrong. It's unfortunate that pixhawks are now being sold that deviate from the standard.

If you have a GPS fix you can manually run fake_mag which publishes another sensor_mag instance populated with the expected magnetic field data for your location and attitude. This will only work if the heading is roughly correct. If you get a log rotating the vehicle on different sides (kind of like the mag cal dance, but don't worry about being thorough) it will be obvious how to fix both your mags.

RescueBeaver commented 3 years ago

Thanks! I`ll try tomorrow.

And I have one more another Pixhawk, but there is not internal mag.

le-chat commented 3 years ago

@RescueBeaver wrote a log with rotation around three axes: https://logs.px4.io/plot_app?log=01e48f0e-459a-47ac-a185-cf14479cefa4

For MAG1, we see from comparison of gyro data (vehicle_angular_velocity/xyz.{00,01,02}) and sensor_mag.01/{x,y,z}:

(This includes some guessing, because of noisy data and few data points.)

That means X-axis of magnetometer is Z of aircraft and Y-axis of magnetometer is reversed to X of aircraft. (So Z of magnetometer is a reversed Y of aircraft.) This seems to be ROTATION_ROLL_90_PITCH_180_YAW_90.

For MAG0 I couldn't perform such analysis, because there are too few points in sensor_mag.00/{x,y,z} and they are too messy, probably we need more detailed SDLOG_PROFILE.

le-chat commented 3 years ago

@RescueBeaver wrote a log with rotation around three axes: https://logs.px4.io/plot_app?log=01e48f0e-459a-47ac-a185-cf14479cefa4

That means X-axis of magnetometer is Z of aircraft and Y-axis of magnetometer is reversed to X of aircraft. (So Z of magnetometer is a reversed Y of aircraft.) This seems to be ROTATION_ROLL_90_PITCH_180_YAW_90.

This was incorrect, because in the log above CAL_MAG1_ROT was already non-zero arbitrary value.

From the log https://logs.px4.io/plot_app?log=f51e9380-914f-4c0e-82ea-dce5742ae8b0 we finally derived ROTATION_ROLL_90_PITCH_90, and this value really works.

Thanks for the help, @dagar ! I hope we will calibrate MAG0 successfully too.

dagar commented 3 years ago

For MAG0 I couldn't perform such analysis, because there are too few points in sensor_mag.00/{x,y,z} and they are too messy, probably we need more detailed SDLOG_PROFILE.

You can turn on "Sensor comparison" (Bit 6) in SDLOG_PROFILE

dagar commented 3 years ago

@RescueBeaver wrote a log with rotation around three axes: https://logs.px4.io/plot_app?log=01e48f0e-459a-47ac-a185-cf14479cefa4 That means X-axis of magnetometer is Z of aircraft and Y-axis of magnetometer is reversed to X of aircraft. (So Z of magnetometer is a reversed Y of aircraft.) This seems to be ROTATION_ROLL_90_PITCH_180_YAW_90.

This was incorrect, because in the log above CAL_MAG1_ROT was already non-zero arbitrary value.

From the log https://logs.px4.io/plot_app?log=f51e9380-914f-4c0e-82ea-dce5742ae8b0 we finally derived ROTATION_ROLL_90_PITCH_90, and this value really works.

Thanks for the help, @dagar ! I hope we will calibrate MAG0 successfully too.

I'm glad you got it working and I apologize for the inconvenience. Now that we've got the legacy rotation mess cleaned up things like this will be a lot more consistent going forward.

dagar commented 3 years ago

@mengchaoheng @rizvanovic feel free to open a new issues with logs if you're still having problems.