ArduPilot / ardupilot

ArduPlane, ArduCopter, ArduRover, ArduSub source
http://ardupilot.org/
GNU General Public License v3.0
10.78k stars 17.22k forks source link

Pixhawk 6c mini external compass disappearing #25946

Closed ashwanisihag closed 4 months ago

ashwanisihag commented 8 months ago

When we use pixhawk 6c mini with m9 or m10 the external compass disappeard after few flights. Its been reported in almost every system that external compass is missing. Internal is always detected but external disappears. May be because they are same Ic.

We have checked everything and all seems normal when we reinstall firmware. Its not a loose connection problem.

boba1234567890 commented 7 months ago

Hi I've also been having this problem but with my Pixhawk reinstalling the Sofware (ArduCopter v4.4.4) is not solving anything. even downgrading to (v4.4.3) it still doesn't work.

ashwanisihag commented 7 months ago

Here is Full description to duplicate the issue and solution:

Pixhawk 6C Mini and M10 or M9 both has same compass IST8310 Compass. So after few flights external will be not available or not detected or some thing goes wrong when we have same ic as external and internal as compass.

Solution: Not reliable but restarts works sometime but not always.

rmackay9 commented 7 months ago

@ashwanisihag, @boba1234567890,

Can you tell us exactly which compass module you are using? Is it part of a GPS unit? Perhaps provide a picture if possible. Also if you have an onboard log that would be great. You should be able to create a log even if the autopilot is not armed if you first set LOG_DISARMED = 1.

rmackay9 commented 6 months ago

@ashwanisihag @boba1234567890,

Sorry to pester but could you clarify which compass module is being used?

rmackay9 commented 6 months ago

Hi, no feedback so I think I'll close this issue now. If we do hear back from @ashwanisihag or @boba1234567890 then I'm very happy to re-open.

ashwanisihag commented 6 months ago

pix Hi Randy

Sorry I missed your message was bit busy with army..thanks for replying...Please find the photo attached above of the hardware we are using. Its a big problem now as we have bought 1000 FC units and all have the same issue.

By the way its pixawk 6c mini and M9 or M10 gps units from holybro. After few restarts it works fine but when on training with army external compass sometimes disappears on ground and its frustrating.

Because the external compass disappers it says "Compass inconsistent"

If this is fixed then I will test 4.5.0 today and will report you asap because it comes rendomly and not so easy to duplicate.

Regards

ashwanisihag commented 6 months ago

@ashwanisihag, @boba1234567890,

Can you tell us exactly which compass module you are using? Is it part of a GPS unit? Perhaps provide a picture if possible. Also if you have an onboard log that would be great. You should be able to create a log even if the autopilot is not armed if you first set LOG_DISARMED = 1.

HI compass is IST8310 in M9N or M10N Pixhawk 6c Mini has also the same compass onboard

ashwanisihag commented 6 months ago

Just tested

Same issue in 4.5 beta Screenshot 2024-03-17 161625 1 01-01-1970 05-30-00.zip

ashwanisihag commented 6 months ago

Is this the issue?

"The IST8310 is more complicated to auto detect because it has 4 different I2C slave addresses." https://intofpv.com/t-discover-compass-ist8310-in-betaflight Although its related to betaflight code

rmackay9 commented 6 months ago

Hi @ashwanisihag,

Thanks for the feedback and links, I've re-opened the issue and hopefully we can reproduce it and/or just fix it.

tridge commented 6 months ago

@ashwanisihag we do look for 4 different i2c addresses for probing: https://github.com/ArduPilot/ardupilot/blob/Plane-4.5/libraries/AP_Compass/AP_Compass.cpp#L1232 if you have spare boards that reproduce this issue reliably you could send one to me otherwise we may need Saleae pro traces of the i2c bus

tridge commented 6 months ago

@ashwanisihag feel free to ping me on discord too

ashwanisihag commented 6 months ago

@ashwanisihag we do look for 4 different i2c addresses for probing: https://github.com/ArduPilot/ardupilot/blob/Plane-4.5/libraries/AP_Compass/AP_Compass.cpp#L1232 if you have spare boards that reproduce this issue reliably you could send one to me otherwise we may need Saleae pro traces of the i2c bus

Yes, I can send you one set. Please guide me how to send and at what address.

ashwanisihag commented 6 months ago

@ashwanisihag feel free to ping me on discord too

Sure i will try

ashwanisihag commented 6 months ago

@ashwanisihag we do look for 4 different i2c addresses for probing: https://github.com/ArduPilot/ardupilot/blob/Plane-4.5/libraries/AP_Compass/AP_Compass.cpp#L1232 if you have spare boards that reproduce this issue reliably you could send one to me otherwise we may need Saleae pro traces of the i2c bus

I have seen this code and tried to figure out but was not abe to figure it out.

rmackay9 commented 6 months ago

@ashwanisihag,

I'd like to get to the bottom of this if possible. Could you ping Andrew Tridgell or me on Discord? I think maybe your Discord id is different from your GitHub id?

rmackay9 commented 5 months ago

No updates for a week so I'll close this issue again. We will see responses however.

ashwanisihag commented 5 months ago

No updates for a week so I'll close this issue again. We will see responses however.

I will create Discord id and will connect. Thanks

rmackay9 commented 5 months ago

I've received another ping that this issue is real.

Craig-1965 commented 5 months ago

Hi all.

I have had the same problem with the PH6C Mini

First issue was in a field test, I unfortunately had the internal compass temporarily disabled. The drone had been flying perfectly and was well tuned. All of a sudden it turned 90 degrees and flew off at a speed in excess of 120km/h, never to be seen again.

The next drone I had to disarm and violently crash a flyaway mid flight due to the External compass disappearing an the Satellite connection degrading drastically (simultaneously) it dropped 60% of Connected satellites.

Product support from Holybro was poor and they attempted to place the blame cack on my setup. To add context we have over 250 units in flight using the PH4 Mini and Pix32 from Holybro and no other such issues.

I have tested the setup ad nauseum using the HolyBro M8N, M9N and newest M10N and had intermittent faults on all. Just powering the drone up and letting it stand for an hour,, powered up and along the way the External Compass Disappears and connected satellites will drop fom 18-21 down to 9-11.

Craig-1965 commented 5 months ago

With continued communications with Holybro I received the following Email 11 March 2024 (and nothing since then nothing)

**Hello,

We have found that the lost external compass on a Pixhawk 6c running Ardupilot connected with a M9N GPS is caused by the Ardupilot firmware. We need some time to get it fixed. Now please give it Px4 instead which is free of such an issue. Will keep you updated.

Thanks!

Holybro**

vincentpoont2 commented 5 months ago

With continued communications with Holybro I received the following Email 11 March 2024 (and nothing since then nothing)

**Hello,

We have found that the lost external compass on a Pixhawk 6c running Ardupilot connected with a M9N GPS is caused by the Ardupilot firmware. We need some time to get it fixed. Now please give it Px4 instead which is free of such an issue. Will keep you updated.

Thanks!

Holybro**

This issue indeed only occur with Ardupilot, but sorry that our rep tell you to "give it Px4 instead which is free of such an issue".

We are working with @rmackay9 @tridge to fix this issue in Ardupilot. The hardware are already on the way to tridge for troubleshooting, and we will provide update here when we have a solution.

Vince Holybro

Craig-1965 commented 5 months ago

Doesn't do anything for the units lost and other damages or the approach followed by the Support staff.

I'll keep the other units out of the air until addressed.

What doesn't make sense and as discussed with he support staff. Without a UBEC powering the Servo rail it remails a lot more stable. I have one unit that I personally fly and I have around 22 hours of flying on it without incident. It just so happens that this Servo rail is powered by a 5A UBEC from Tarot and if I change the UBEC to a different brand the instability returns.

Doesn't make sense I know but this is live experience (may or not be of value) and this unit is one of the best drones I have in use at the moment.

rmackay9 commented 5 months ago

@Craig-1965,

If you have a log of the compass failing/disappearing in flight we'd really like to see it. If that can be done without flying it all the better, maybe set LOG_DISARMED = 1.

Craig-1965 commented 5 months ago

It will be IMPOSSIBLE to get the Logs for the ones where it happened in Flight.....

Those 2 Drones, Cameras, Payloads etc are still GONE/LOST/MISSING. They did a sudden direction change and both flew off at 120km/h+ Until the battery ran out and were never seen again. The one went into Alt Hold about 3.5km from home and I managed to fly it back on the screen for about 100m and then it shot away at 90 degrees to flight path and I lost comms at 6.1km over a residential area.... Luckily only travelling at 18m/s when it went down. I haven't seen reports of injury or property damage, but I am sure there was.

I'll just have to get the SD Card from the last used 6C Mini. I will try the LOG_DISARMED = 1 and send the recorded log.

I'm just out of country right now so will probably only get it done at the earliest next week.

tridge commented 4 months ago

@ashwanisihag @Craig-1965 I have the hardware now, sent by @vincentpoont2 - what I need is a way to reproduce the issue. So far I've been testing on USB power, with numerous soft reboots to see if it ever comes up without the compass. I also tried having it sit for a long time to see if the compass would disappear. No luck so far reproducing the issue. do either of you have suggestions on approached to triggering the issue?

Craig-1965 commented 4 months ago

It's a tricky one. I have 1 in a drone that I have now logged 400 flights with no issue and then the 2 that flew away were also working perfectly and then flew away.I have one but it's 1600km away. Will arrange for them to courier the unit to me with a shipment of drones that need to go to Australia after fine tuning.That unit is very badly behavedCraig Neil+27 82 448 7226 -------- Original message --------From: Andrew Tridgell @.> Date: 2024/04/30 05:55 (GMT+02:00) To: ArduPilot/ardupilot @.> Cc: Craig-1965 @.>, Mention @.> Subject: Re: [ArduPilot/ardupilot] Pixhawk 6c mini external compass disappearing (Issue #25946) @ashwanisihag @Craig-1965 I have the hardware now, sent by @vincentpoont2 - what I need is a way to reproduce the issue. So far I've been testing on USB power, with numerous soft reboots to see if it ever comes up without the compass. I also tried having it sit for a long time to see if the compass would disappear. No luck so far reproducing the issue. do either of you have suggestions on approached to triggering the issue?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

tridge commented 4 months ago

@Craig-1965 @ashwanisihag I have now managed to reproduce the issue and I've opened a PR that I think will fix it: https://github.com/ArduPilot/ardupilot/pull/26929 I've put builds of plane and copter 4.5.1 with this fix here: http://uav.tridgell.net/tmp/IST8310/ Please test!

Craig-1965 commented 4 months ago

Cannot access the .apj file

Tried on 3 PC in 2 different networks

From: Andrew Tridgell @.> Sent: Tuesday, April 30, 2024 5:55 AM To: ArduPilot/ardupilot @.> Cc: Craig-1965 @.>; Mention @.> Subject: Re: [ArduPilot/ardupilot] Pixhawk 6c mini external compass disappearing (Issue #25946)

@ashwanisihag https://github.com/ashwanisihag @Craig-1965 https://github.com/Craig-1965 I have the hardware now, sent by @vincentpoont2 https://github.com/vincentpoont2 - what I need is a way to reproduce the issue. So far I've been testing on USB power, with numerous soft reboots to see if it ever comes up without the compass. I also tried having it sit for a long time to see if the compass would disappear. No luck so far reproducing the issue. do either of you have suggestions on approached to triggering the issue?

— Reply to this email directly, view it on GitHub https://github.com/ArduPilot/ardupilot/issues/25946#issuecomment-2084328536 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AQKWSMUEHVK4ABEB7ELUOQDY74IZJAVCNFSM6AAAAABBVDTPDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUGMZDQNJTGY . You are receiving this because you were mentioned. https://github.com/notifications/beacon/AQKWSMU7SP7TYNR2YCIS7ULY74IZJA5CNFSM6AAAAABBVDTPDSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTT4HRKFQ.gif Message ID: @. @.> >

rmackay9 commented 4 months ago

@Craig-1965,

I've checked Tridge's link and there does not appear to be any password required to download from the site. Maybe you could try again with a different browser?

image

tridge commented 4 months ago

@Craig-1965 the problem was likely your browser auto-converting to https and my servers SSL records being corrupt please try this now: https://uav.tridgell.net/tmp/IST8310/ I have also put them here: https://drive.google.com/drive/folders/12VJm_4b4449iJcfqIM1okFV8p19D-_WU?usp=sharing

Craig-1965 commented 4 months ago

Thank you. It might have been a network issue but I returned there shortly after and downloaded the file.Will this fix be included in all future firmware updates for the 6C Mini or should we stick to this firmware?Craig Neil+27 82 448 7226 -------- Original message --------From: Andrew Tridgell @.> Date: 2024/04/30 23:25 (GMT+02:00) To: ArduPilot/ardupilot @.> Cc: Craig-1965 @.>, Mention @.> Subject: Re: [ArduPilot/ardupilot] Pixhawk 6c mini external compass disappearing (Issue #25946) @Craig-1965 the problem was likely your browser auto-converting to https and my servers SSL records being corrupt please try this now: https://uav.tridgell.net/tmp/IST8310/ I have also put them here: https://drive.google.com/drive/folders/12VJm_4b4449iJcfqIM1okFV8p19D-_WU?usp=sharing

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

tridge commented 4 months ago

@Craig-1965 if your testing goes well we will include it in the 4.5.3 release. We did not include in todays 4.5.2-beta as we don't yet have confirmation it fixes your issue. Also note that @vincentpoont2 from Holybro has indicated a hardware fix may be coming