Onboard Mag calibration doesn't complete

[lvale]

Issue details

Onboard mag calibration doesn't complete most of the time.

Used MP latest beta to start and monitor the process.

Tested with the 3 mags on PixRacer, and 2 Mags on PixHawk

While some of the mags complete, others don't complete, so calibration never completes.

On one occasion mag3 completed 21 times, mag1 2 times, and mag2 never completed.

Resorted to calibrate using the old process, and took less than 1 minute.

As this is quite a frequent process (too frequent?) because GCS is always "complaining" about mag problems- pre-arm compass variance- (coming from EKF) this is quite serious, specially because it requires user reboots (power off/power on) to store the values...

Version

3.4Master

Platform

[ ] All [ ] AntennaTracker [X] Copter [ ] Plane [ ] Rover

Airframe type

Quad1 580mm and Quad2 250mm

Hardware type

PixHawk and PixRacer

[OXINARF]

I think this is the old fitness problem. Have you tried increasing the COMPASS_CAL_FIT value? If I'm not mistaken, I think it is worse in Pixracer because it has two internal compasses in a small space.

[lvale]

If so, then the usual manual calibration shouldn't report as complete as it does in less than a minute for either the 3 mag on the racer or the 2 mags on the PH.

[OXINARF]

The manual calibration is done by the GCS while the onboard calibration is done by ArduPilot. That means they can have different levels of acceptance. Increasing the COMPASS_CAL_FIT value will increase the acceptance level of the onboard calibration.

[bugobliterator]

@lvale the onboard compass cal is very precise and does not tolerate calibrating compass in bad magnetic environment, offboard compass cal will calibrate and even give you results but that data has a high chance of being wrong as it does not do such checks. The internal compass cal fits 9 parameters and unlike available offboard compass calibration will not calibrate if a) you are calibrating in magnetically polluted environment b) compass is mounted at a location where there are lot of electromagnetic noise. If the onboard compass calibration is failing it is not recommended to use that compass as primary in flight, also you can increase the COMPASS_CAL_FIT as @OXINARF suggested if you want to use it anyway. Also @OXINARF its not a problem it is how its supposed to be, we need to convey if compass is actually good enough. I think a patch to have a separate tolerances for each compass instance would be a nice to have.

[bugobliterator]

https://github.com/ArduPilot/ardupilot/pull/4738 sort of helps with this situation.

[magicrub]

Is there a way to know if there is EM noise making the calibration from happening? If so, that should be shown to they user somehow.

Also, should the default fit value be different for the pixracer?

On Aug 28, 2016 7:24 AM, "Siddharth Bharat Purohit" < notifications@github.com> wrote:

4738 https://github.com/ArduPilot/ardupilot/pull/4738 sort of helps

with this situation.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ArduPilot/ardupilot/issues/4734#issuecomment-242977613, or mute the thread https://github.com/notifications/unsubscribe-auth/AEj7G9E5_72Av5kZ4zZ5DMVBCmlQJmscks5qkZoJgaJpZM4JuwX1 .

[bugobliterator]

@magicrub the fitness result is returned after the calibration is done to GCS, that is something that could definitely be interpreted and shown to user. Regarding the pixracer if the compass is internally mounted that is a possibility, but in that scenario one shouldn't use the internal compasses of pixracer as primary and loosen up fitness criterion for extra internal compasses through different parameter, as made possible by above mentioned PR.

[DonLakeFlyer]

I'm sure some of you follow the PixRacer RCGroups forum. In there you'll find many, many reports of folks not being able to use ArduPilot on a pixracer because nobody can figure out how to get past the compass cal/variance problems. Most people including myself have given up and just always run PX4 Pro on PixRacer. Having to set a plethora of mysterious parameters to make it work correctly just isn't going to fix the problem for folks that don't have to time to understand the details of how the internals work.

[bugobliterator]

@DonLakeFlyer I understand your frustration, if the above patch or something similar (that lets user set fitness value for different compasses) makes in and proper display and ability to set them on the calibration page of GCSs could be really helpful. Also making use of the information that is provided to GCS via mavlink message. I think you are a very major contributor to QGC, so If you need any sort of understanding or help from my end in making this interface seamless for users, I would be more than happy to give that to you. The same goes to APM_Planner lead dev @billbonney and MissionPlanner lead dev @meee1 .

[DonLakeFlyer]

I'm just not sure giving users more dials to twiddle is the way to solve this problem. Users just won't figure that out. I think it needs to be way more automatic than that.

Also given what I'm reading here it almost seems like the firmware code has changed to the point where using the new onboard compass cal is a requirement in order to get things to work. Since it sets additional things beyond just the offset values. That also seems like a new thing. Is that correct?

I'm concerned that compass calibration correctness requirements have gone so high that the ROI for I guess better compass readings isn't worth the effort. People are bailing out of ArduPilot because of it.

[DonLakeFlyer]

http://www.rcgroups.com/forums/search.php?searchid=57624230&query=compass

[bugobliterator]

I'm just not sure giving users more dials to twiddle is the way to solve this problem. Users just won't figure that out. I think it needs to be way more automatic than that.

@DonLakeFlyer its going to be just one at least for now i.e. COMPASS_CAL_FIT parameter and only one value to display i.e. fitness. Which I think is entirely reasonable.

@lvale can you post a link to log file for the compass variance problem you are facing?

I'm concerned that compass calibration correctness requirements have gone so high that the ROI for I guess better compass readings isn't worth the effort. People are bailing out of ArduPilot because of it.

The requirement bar can be tweaked by doing just one parameter tweak COMPASS_CAL_FIT which is mentioned above and user should be told that this is last resort and that he must understand that the compass being used might be bad or in a bad environment. But after increasing the value above the Fitness value received from last failure will let the calibration move forward and if EKF is happy user can arm and fly the copter.

[lvale]

@bugobliterator Let me try to locate the appropriate logs (I might try to redo all over again), BUT

please note that the onboard compass cal fails with either a PixRacer AND a PixHawk.

On the PixRacer one of the internal mags manages to calibrate, the other doesn't.

Calibration was performed on the usual place I usually do, so permanent stuff would be the same

[bugobliterator]

@lvale since you are doing this test, is it possible to get a table mentioning the resultant fitnesses of the calibrations you've tried?

[lvale]

sure

[bugobliterator]

@lvale thanks!

[magicrub]

Is px4 making it easier on the pixracer by not using the second compass? What's the purpose of having two on the pcb anyway?

On Aug 28, 2016 12:47 PM, "Siddharth Bharat Purohit" < notifications@github.com> wrote:

@lvale https://github.com/lvale thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArduPilot/ardupilot/issues/4734#issuecomment-242995103, or mute the thread https://github.com/notifications/unsubscribe-auth/AEj7G4dhr6pvifvMsWtkqzmMxm9gLDLOks5qkeWxgaJpZM4JuwX1 .

[lvale]

2 logs both completed https://drive.google.com/open?id=0BzRxu4IwHPYCTHdYcWNyZGE4eFE

[OXINARF]

It was clear months ago that the current fitness value is giving Pixracer users too much trouble. We wanted that people tried with different fitness values but that didn't happen.

Like Don said we need to make things easier for users or they will leave ArduPilot.

[lvale]

@OXINARF The logs are from a PixRacer but I experienced similar results with only the 2 mags of a typical PixHawk system. And the final power off/power on must also be revisited :)

[proficnc]

some more mods are likely here as PH2 can have up to 9 mags show up or more. clearly we will need to disable the unwanted ones.

[DonLakeFlyer]

How does the user have any idea what to set COMPASS_CAL_FIT to? Just keep changing it until it works? If that's the answer why can't compass cal figure that out itself and then tell the user they have a crappy compass and the settings have been adjusted appropriately to make things work. If they are still left with one good compass they are good to go. I know I'm being a pain but I would love to see vehicles running ardupilot become usable by people that don't need to understand magic settings.

[wjax]

Compasses problems are being constant in the community since the EKF era. Compass variance is happening a lot in every copter I see around and forums are full of this.

I have done tests with a bare pixhawk sit in a table, calibrate and received the same EKF compass variance errors. That lead to think that compass calibration in gcs is not good.

It seems that with 3.4 a new onboard calibration is available. And with it, soft iron and ellipsoid fit compensation that were not present in version 3.3 AFAIK.

With onboard calibration I have been able to avoid EKF compass variance errors most of the time on ground while manually rotating 360. So some improvement is real.

CAL_FIT is the way to go but if the compass receives lot of interference it will not calibrate even with a CAL_FIT of 20. The thing is if it is OK to not calibrate as it shows a potential problem or if the problem does not exist at all... Hard to say.

What it is true is that for me and others 3.2.1 keeps being our daily arducopter firmware.

[jschall]

@DonLakeFlyer COMPASS_CAL_FIT was not originally intended to be a parameter. It ended up needing to be because some boards had compass noise issues.

It is intentionally tight so that you cannot screw up the calibration by moving the vehicle through external magnetic fields.

[rmackay9]

I've added an issue for MP to add an easier method for users to set the compass-cal-fit parameter. I'm not saying this is a full solution of course, it's just something small that I can think of to help the situation a bit. https://github.com/ArduPilot/MissionPlanner/issues/1355

[lvale]

Reopened for further discussion

[lvale]

Further to our discussion, I have a pixracer on the bench (so no big metal parts around) and did a onboard compass calibration wired via USB. The primary compass is on the GPS board (5883?)

I'm attaching the resulting bin and the QGC log (tlog?)

And the PreArm:Inconsistent compasses is sent to the GCS

calibration logs.zip

[OXINARF]

Can you try it without USB? I seem to recall @jschall saying it would be worse when using USB in the Pixracer. Also, I think the conclusion about this was that the mags in the Pixracer weren't in good places in the board so it would never get great results.

[DonLakeFlyer]

Yeah, I've found that doing while connected via USB really generates crappy values with the PixRacer. Hence the need for this to work over WiFi.

[rmackay9]

We've relaxed the compass calibration "fitness" for all boards so I think this issue is resolved. This will go out to users with AC3.5.
The problems doing calibration over Wifi is already raised as a separate issue. https://github.com/ArduPilot/ardupilot/issues/5505

Closing OK?

[andyp1per]

What's the magic value? I am seeing this on 3.4.4 with my external ublox compass (internal AUAV-X2 calibrates fine weirdly). It's no good disallowing compass use in high mag environments - my QAV 250 is an EMF nightmare, but I have no option for moving stuff away, I need the compass to just work - and it pretty much did in 3.3.

[andyp1per]

Just as an update, I needed a COMPASS_CAL_FIT of 12 and my external then calibrated with a fit of 11.2 and my internal with a fit of 8.2

[maciek01]

I know its a dead thread, but just trying. I have the same problem on a big rig - tarot hex 680 with Pixhawk Cube and Here 2. I spent endless hours twisting and turning the HEX all around my backyard with the heavy battery on - at least 3-4kg to no avail. I have relaxed the fitness to minimum. Also put the same HERE2 gps on much smaller QUAD where it calibrated well. All i ended up with is severe muscle pain overnight. There has to be a better way - DJI figured it out.

[rmackay9]

@maciek01, if it worked on a small quad then it is perhaps likely that there is a lot of metal near the compass. It may help to disable the internal compasses if the flight controller can't be moved away from metal. Another reason why the compass calibration can fail is because the eventual offsets are unexpectedly large. Try increasing the COMPASS_OFFS_MAX. Try increasing this to has high as 1800 to see if that helps.

One challenge AP faces is that we support a massively wide variety of vehicles of varying quality. Getting the balance right between restrictive enough to catch poor builds while being relaxed enough to not cause false positives is a real challenge.

[fnoop]

@maciek01 For the future, there is automatic compass learning in place for the next version: https://discuss.ardupilot.org/t/testers-needed-for-in-flight-compass-learning/34184

[maciek01]

@rmackay9 - thanks Randy, completely understood the point about the wide variety of hardware and pardon my frustration as i have been struggling with this now for months. Awkward thing though is that if i put back HERE (not HERE2) things calibrate fine, as they have been so for the last 2 years on this HEX. The same HERE2 works well on smaller rigs. I will play with the parameter you suggested. And, yes - all internal compasses are off - learned that lesson during the initial build of this HEX.

[maciek01]

@rmackay9 I tried COMPASS_OFF_MAX as high as 3000 and still no luck. Just want to mention that there is nothing special about this build -here is the harware list:

frame - tarot 680 bat: tattu 16000 mah 6S motors: tarot 380 props: off gimbal - tarot gopro hero 4 lidar lite v3 pix4flow pixhawk 2.1 here2 standard aluminum gps mast from ebay

COMPASS_CAL_FIT = 32 COMPASS_OFFS_MAX = 3000

no issues calibrating Here (1)

I have 2 x Here2 - they both calibrate fine on a smaller 500 quad and both fail on this hex

at this point i am at a loss - considering going back to Here1 - less than ideal

[rmackay9]

@maciek01, can you try setting the COMPASS_CAL_FIT = 64?

[maciek01]

@rmackay9 i tried with both 64 and 100. The same negative result. BTW, MP gives a warning that this parameter value is out of range.

Is there some kind of compass calibration log i could activate and attach for review? completely blind as to what the cause may be.

thanks! m.

[rmackay9]

@maciek01,

I think that it's most likely that because the calibration completes successfully when the flight controller is detached from the frame that there's some piece of metal in the frame near the flight controller that is causing troubles. Perhaps some metal and/or magnetic screws that have such a strong influence that they're distorting the magnetic field so badly that the calibration won't pass no matter how high the cal-fit parameter is set. I'm a little surprised, I've never found a situation myself that was this bad.

It's possible that if a tlog is posted that someone on the dev team will look at it but really, it's most likely not a software bug but rather a situation that is very far outside the normal tolerances. A hardware solution is probably best..

[maciek01]

Thanks @rmackay9 . Not sure if i can get tlogs as this calibration is done via on board and triggered by radio, The only reason i would question the metal object theory is that Here 1 has no issues during calibration.

tarot is a carbon fiber based frame. i have a small PDB far below - 25 cm below Here 2. All internal compasses are disconnected.

[maciek01]

@rmackay9 so as i suspected, when i replaced here2 with here1 on this rig calibration procedure succeeded. The only difference i can think of is that this is an older PH 2.1 - purchased DEC 2017 versus OCT 2018 on the other one that calibrates fine with here2.

i hope its not an unpublished hardware incompatibility issue.

[66nola66]

Does anybody know the physical location of the mag on the pixhawk board??