flight controller maxing out one motor causes vehicle to go down

[expntly]

Issue details

I've seen this happen over and over. One motor maxes out its PWM range, its diametrically opposed sibling goes to MIN, vehicle loses altitude, resulting in very hard landings (10G on this last instance). Meanwhile, the other motors have quite some margin and aren't using it.

There definitely seems to be some CCW yaw in the frame, but CCW avg is 1784, and CW avg is 1377, so there's some room to go.

I've usually traced this issue back to either lack of thrust, or bad ESC programming. This vehicle was empty and has close to 2x the thrust compared to all up weight. We checked the ESC and there's one (ESC#8) that had a slightly different config but that shouldn't have any effect on the performance.

Version

AP3.8.5beta1 (though this has been happening for over a year)

Platform

[ ] All [ ] AntennaTracker [ ] Copter [X] Plane [ ] Rover [ ] Submarine

Airframe type

OctaQuadplane X8

Hardware type

PH2.1 (seen this on PH1 too)

Logs

https://drive.google.com/open?id=1zsuOGRnK1hG25LNJH1NBeLBZs9z_rAWJ

[rmackay9]

Thanks for the report. This is really a duplicate of these issues:

https://github.com/ArduPilot/ardupilot/issues/7525 https://github.com/ArduPilot/ardupilot/issues/838

If this were a quadcopter what is shown in the graph above is the best we can do because the vehicle needs to sacrifice either attitude control or altitude and altitude is certainly the best choice. In the case of a hexacopter though we have the option to sacrifice yaw control only.

Anyway, the two issues linked above capture what needs doing so I'll close this item.

[expntly]

It seems like you're suggesting we had a motor failure?

As an X8 we've had motor failures before (missing more than a single motor) and landed no problem. The logs did not show this pattern.

[rmackay9]

@expntly,

Yes, I think there's a significant mechanical issue probably with Motor/ESC 7. It could be a very badly calibrated ESC, a motor failure, a lose propeller or something else similar. This leads to a very bad motor imbalance with and shows this classic pattern of a single motor going high and the opposite motor going low.

As mentioned above, there are some improvements we can make in the code to handle it better (detect and remove the motor from the limiting) but there is definitely a hardware issue here. If the vehicle is in a stable hover, most vehicles will have all motors within 150 pwm of each other.

If you have a previous log of a motor failure on an X8, I'm pretty sure we will see this same pattern.

[expntly]

We have lots of such instances and I'm going through the logs, but here's an interesting one where a bad prop resulted in this This is the front left corner of an X8, i.e. motor 2 (C6) and 5 (C9), with 5 being the one which prop initially broke apart when de-transitioning, and this also damaged the blades of motor 2. The vehicle landed no problem (good job FC)!

Here's the DF log, look near the de-transition at 18:20 or so: https://drive.google.com/file/d/1a5gfGyTlgmGWJpvMmHmlmVR2PzPRKL5N/view?usp=sharing

Two things I'm seeing when looking at RCOU.C5 to C12:

• C6 and C12 are hyper-frequently maxed out at the time of the incident, yet no other motor is anywhere near MIN
• when the VTOL descent starts, the motors leave this initial phase of PWM frenzy and things seem to be back to normal, yet we're missing 100% of motor 5's prop, and a good chunk of motor 2's prop

I'll try to look for more examples if you're interested.

[expntly]

@rmackay9 Also, what is it that makes you think Motor/ESC 7 has an issue? This is RCOU.C11, correct? I don't see what's wrong with this motor from the logs, maybe I'm missing something.

[lthall]

Randy is saying that the motor connected to RCOU.C7 has a serious problem.

You may be getting into an overheat shutdown or a low voltage shutdown on that ESC.

In all these cases we are dealing with a mechanical issue that should be fixed. No aircraft should rely on the code to deal with a physical problem.. From a dev's perspective we add handling for lost motors and other physical problems to ensure the pilot has every opportunity to fix these problems and minimise the risk when they happen, not so they can knowingly fly with them.

[rmackay9]

It is possible the opposite motor might not hit the minimum if the average throttle required to hover is high enough.

I guess what I'm saying is when we see the pattern of a single motor going high and its opposite going low, it is very very likely to be a motor/ESC/propeller failure. We're not saying that this is the only pattern for a motor/esc/propeller failure though.

[expntly]

We just had another vehicle have the same issue, except this time at least two motors (rear-top-left as RCOU.C7 and RCOU.C11 and rear-bottom-right) went idle right at de-transition. This is a smaller vehicle but with the same X8 propulsion.

DF log: https://drive.google.com/open?id=1C5sNJq9ywKO05EX-4kForl48p9oR0rea

We're not ruling out hardware problems, but again this is a different vehicle. Actually this has happened with more than half a dozen vehicles we've had for the last year or so.

By any chance, are there any known incompatibilities with AP and KDE ESCs? This is the one we've used recently: https://www.kdedirect.com/collections/uas-multi-rotor-electronics/products/kdexf-uas95hvc

[DarkmatterVale]

@expntly @rmackay9

I am part of a team competing in the AUVSI SUAS competition and we developed a custom, high-payload capacity octocopter with a thrust to weight ratio of 2.7:1. We thought that we had developed a failure-resistant UAV, using redundant electrical and sensor systems. Unfortunately, today, we had a fatal crash. In this crash, it appears we had a single motor failure occur followed by a complete loss of altitude control on the UAV.

In the DF and Telemetry logs (attached to this post), it can be seen that a single motor fails at 11:37:37. At this point, the UAV attempts to lock yaw, resulting in another motor to drop its thrust. The UAV begins losing altitude, at a higher and higher rate. Even after being switched into loiter and providing 100% throttle input, the UAV continues to lose altitude. We were shocked to see the UAV crash, because we believed the high-thrust propulsion system enabled us to lose a single . We did a thorough analysis of the UAV's components post crash, and verified that there were no shorts or any damage incurred prior to the crash. Also, it is important to note, that although we had a compass EKF, it was very short and subsequently returned to normal (and thus we have determined that was not a cause of the crash). We are at a loss as to what could have caused the crash. Could it have been an issue with the Pixhawk not being able to maintain the UAV's altitude?

Here is key information regarding our UAV's specs: 1) Weight: 28.22 lbs 2) UAV Configuration: Octocopter (flat) 3) Motors: KDE4213XF (https://www.kdedirect.com/collections/uas-multi-rotor-brushless-motors/products/kde4213xf-360) 4) Propellers: KDE 18.5" x 6.3" dual, folding CF propeller 5) ESCs: UAS55 (https://www.kdedirect.com/collections/uas-multi-rotor-electronics/products/kdexf-uas55) 6) Autopilot: Pixhawk 2.1 (standard, dual HERE modules) 7) Current/Voltage monitoring: MAUCH

Please let me know if I can provide any additional data to assist in further analysis.

Links to the logs: 1) TLog: https://drive.google.com/drive/folders/1TJMrEv7uRYenCJZ3d4445r-n1GyjY6LV?usp=sharing 2) DF log: https://drive.google.com/file/d/1or-TIa0ksydIVhvm7e8OJFJow57z2FOS/view?usp=sharing

[expntly]

Yup, we're in the same boat, sorry :-(

First of all, it's striking how similar our stacks are: KDE + PH2.1 + MAUCH. That's suspicious. We're on AP though, not AC, but I think they use the same motor mixing code. Are we two the only KDE users out here? We're not super happy with the hardware, it's just never worked very reliably (burned a lot of motors, ESCs, and have never really been able to get things stable, with ESCs/motors randomly going idle in flight). The KDE guys are responsive most of the time, but that doesn't fix the hardware.

It looks like you're using MOT_PWM_TYPE=1 (oneshot) and RC_SPEED=490. Looking at your logs, the PWMs vary super (too?) rapidly -- that seems wrong. I remember seeing this before. I think KDE recommends 400Hz, so you could try that. We use PWM_TYPE=0 (oneshot was always worse than normal for us) and 150Hz. We're also seeing issues with these params though, just different ones.

What ESC firmware config/params do you use? We've seen that these can change by themselves and that if one of them differs from the other ESCs then the vehicle will be very unstable. Sometimes some parameters are even missing when we read them back from the ESCs, while we know we did write them at setup time (KDE Direct Device manager doesn't even let you select "no value" anyway).

Do you know if you actually lost the C2 motor? It is the one that's maxed out in the logs.

We have an X8 config and so when we "lose" one motor, the diametrically opposed one goes completely idle (apparently by design), so we effectively "lose" two motors. What's interesting is that in your case this doesn't happen -- C5 goes to ~1300us but that's not your min and it's a CCW motor, not CW like C2. I'm guessing this is because you have an octo+ flat config. That should make your frame more resilient to motor failures? I wonder what @lthall will say.

[DarkmatterVale]

Thanks for the in-depth response. Yes, our UAV is in an octo+ flat configuration, which does make our UAV resilient to motor failures.

You are correct in assuming that the motor PWMs vary too rapidly. After we rebuilt from the crash, we attempted to fly with 7 motors and the platform was very, very unstable. Turns out we had a very bad autotune. We redid autotuning (at the maximum aggressiveness) in zero wind, and the craft performed much better. In the crash, once we lost motor C2, the craft started slowly accelerating down, eventually causing the UAV to enter VRS. At that point, we had lost control of the UAV and could not exit VRS, leading to the crash. It wasn't directly caused by the Pixhawk 2.1 or the firmware. If you are crashing with a similar issue, I would suspect that it is because you do not have enough extra thrust to handle a motor loss or a bad tune.

However, that does not explain why motor C2 failed in the first place. We confirmed that this motor was not spinning at the time of the crash. We thought we had a broken ESC, so we replaced it with a brand new KDE UAS55. To our surprise, during a hover test after completing the autotune, a different motor stopped! This time we were lucky to be close enough to the UAV to hear the ESC making weird noises. It sounded like the ESC had rebooted, and was waiting for a rearm signal. Upon landing, the ESC returned to normal. We then completed a spin test, and to our confusion, the ESC worked fine. When we proceeded to take the UAV back up in the air, the motor was spinning once again.

Long story short, it turns out that the tune was the problem for falling out of the sky. However, we have a very high T:W ratio (2.7:1 at max), which provides us a lot of "buffer" in the event of a motor failure.

Because our UAV is very expensive, we decided to not take the risk anymore and removed the KDE ESCs. We are now using ZTW Spider ESCs, and have not had a problem since. We have narrowed the problem down to the KDE ESCs, and I would recommend not using them. For their price, they do not appear to be very reliable...

[rmackay9]

To add a tiny bit of extra info, it looks like it flew for about 40 seconds with the failed motor before crashing..

[expntly]

@DarkmatterVale

We're also phasing out KDE ESCs. I assume you're using a hybrid of ZTW Spider ESCs and KDE motors/props? How's this worked so far? We're so concerned with KDE that we're thinking about swapping the motors/props too.

How long had you been flying with the KDE stack? Had you seen issues before this fatal crash? And I'm assuming you've only been flying with the new setup for a few days?

[DarkmatterVale]

@expntly

Yes, we are currently using KDE motors and propellers. Thus far, they have worked really well. The motors seem to be very high quality, and we have not had any issues with them thus far. The propellers are very well balanced, leading to low vibrations in flight. We plan on sticking with the KDE motors and propellers.

Prior to the crash, we had not seen any issues with the KDE ESCs. We had flown the UAV for roughly 8 hours in total, all of which was great flying without any problems.

Since we made the switch to the Spider ESCs, I think we have flown ~2 hours. However, we currently do not have KDE propellers on (we broke them all in the crash, waiting for a new set to come in...).

We are looking into getting the UVC line of KDE ESCs, since they will soon have logging capabilities. Unfortunately, we currently do not have the funding to support such an expensive experiment, so it has been placed on the back burner for the next few months.

[expntly]

@rmackay9 @proficnc By the way, the Pixhawk2.1 or Ardupilot don't do anything weird with the servo rail not being powered, right? Or put differently, is there anything wrong with not powering the servo rail?

In our system, there's nothing hooked up to the servo rail except for signal pins. We power everything (ESCs, servo) separately with their own power supply. All grounds should be shared.

[DarkmatterVale]

@expntly KDE recommends you provide the ESCs with a UBEC, as communicated at https://cdn.shopify.com/s/files/1/0496/8205/files/KDE_Direct_UAS_ESC_-_Pixhawk_2.1_CUBE_Instructions_rev2.pdf?8691036834371109334 . However, different ESCs have different requirements. From the Pixhawk 2.1's side of things, it shouldn't require anything.

[expntly]

@DarkmatterVale Yup, we just have a different power system with two MAUCH modules powering a board that maps to ESCs, and that board is powered separately.

Do you use their UBEC?

[DarkmatterVale]

Ah, ok.

No, we do not use KDE's UBEC. We use the MAUCH power cube and run redundant +5V to the servo rail.