ESC stops responding midflight, FW 32.61

[dabit20]

I seem to have some issues with BLHeli32.6.1. Betaflight 4.0, Aikon AK32 6S ESC's, 2x470uF low-ESR cap on the power rails, 2450kv Hypetrain motors, 5S battery, motor_output_limit = 75 in Betaflight, bidirectional DShot enabled.

When I apply full throttle it happens once in a while that an ESC loses it and the quad tumbles from the sky. Once on the ground I can re-arm and continue flight (when no damage prevents that)

Tried reflashing the ESC, advancing the timing to 23 degrees, setting demag compensation to high, using 24kHz/32kHz/48kHz PWM frequencies, increased idle percentage in BF (although it does not happen on motors going idle), switch from DShot1200 to DShot600. It did not solve the issue. Battery voltage sags to 16.4V when it happens, which is not extremely low.

Went back to 32.6.7 (I just had that FW version on my PC), and I could not trigger the error within 2 batteries. With 32.6.1 it takes me 10 seconds; a hover and a punchout suffices.

[sskaug]

Thank you for a good report. I've been looking through the changes from 32.6.7 to 32.6.1, and the prime suspect for potentially causing such a difference, is a modified stall protection. If you'd be willing to test, I could make a test code to investigate this. But then I would need the exact name of your AK32 code (there are several....).

[dabit20]

Hi sskaug,

In the meantime I flew a couple more batteries with 32.67, and I have not seen more mid-flight stalls of a motor. They still do happen when I hit something such as a ghost branch.

I am willing to test of course. The exact name of the firmware I am currently running is Aikon_AK32_35A_Multi_32_67.Hex. Is that enough, or do I need to obtain more info from BLHeli Suite?

[sskaug]

Now there's a test code here: https://github.com/bitdump/BLHeli/tree/master/BLHeli_32%20ARM/BLHeli_32%20Test%20code%20Rev32.6.1%20hex%20files/Test%20code This code has a similar stall protection as 32.6.7.

Apart from this, for a 2450kv at 5S, it may be beneficial to reduce the rampup power setting, in order to reduce the stress on the system during fast WOT events.

[dabit20]

OK, flashed the ESC's to the new firmware. Will try to test as soon as possible. One slight issue: when I flash all of them at once only the first gets revision 32.62, the others report 32.6. I am quite sure that the actual flashing works fine; they were at 32.67 so they received new code. But to be sure I flashed them individually, and then they all report 32.62

I have the rampup power at 25% and the current protection at 60Amp. This seems to keep the MOSFET's alive. The last two ESC's I destroyed suffer from a dead microcontroller. (3.3V drops to ~1.6V, uC becomes hot). Irrepairable damage with BLHeli32 due to the uC serial number verification process unfortunately.

Would you suggest lower values for rampup power than 25%?

5S and 2450kv is a transient situation BTW. I recycled the cells of a couple of 4S packs with issues to 5S packs, and both those packs and the 2450kv motors probably won't last the entire summer.

[dabit20]

Went out to a nearby grass field and flew one pack hard (1300mAh/5S empty in less than 3 minutes). No time for more than that today, unfortunately. Several full throttle runs (which is still only 75% output due to the motor_output_limit set in BF), 180-turns with high throttle applied, and punchouts followed by zero throttle rolling and pitching. I even threw in clipping a tree branch to see if that causes failure.

So far I had no desyncs at full throttle, behaviour is the same as 32.67. When clipping the tree branch behaviour seems better; no out of control spinning as a result. But that might as well be just luck.

[ExaflopS]

@sskaug I have been testing 32.61 on TEKKO32_F3_Metal_Multi_32_61.Hex and Airbot_Wraith32_ST_Multi_32_61.Hex . I have noticed there is issue with event of crashing or hitting branch and motor stopped spinning midair, I have even tried to race, but slightly hitting gate resulting to block motor completly. also when I land on tree and want to try to free quad, the stall protection blocks motors and quopa si useless- also when on ground, turned upside down wanting to use quopa , this blocks motors only by props touching grass. If You need, I can also test if You could make hex files for mentioned escs.

[sskaug]

Thank you both for the feedback and testing. I have now made test codes for the Wraith and the TEKKO. So this seems to be clearly about the compromise between tolerating some blocking and protection of motor/ESC. Going forward, I'll try to see if I can find a more optimal compromise.

[ExaflopS]

@sskaug Thank You, I will test new test blheli FW as soon as possible. I think you are right, protection in 32.61 seems to be wery sensitive. I have few suggestions: Of course there should be found optimal compromise between protection ant tolerating blocking, and that should be default setting and maybe put some configuring option into blheli suite to set it up - for example stall protection: sensitive, standard, weak. Also in order for rescuing quads from high trees, after this protection kicking in disabling motor/esc to allow motor spin up again after some time.

[ExaflopS]

@sskaug I have tested Yours test FW 32.62 - on Airbot_Wraith32_ST_Multi it works normally, I havent found any issue so far, but on TEKKO32_F3_Metal_Multi_32_62 I think it could be problem with rough and weird noises from motors- Before the quad flew without any problem and after flashing 32.62 this is happening- sometimes i can fly without issue and sometimes this weird behaviour is happening all the time: video: https://youtu.be/9G6acWATAEA?t=10

[sskaug]

@ExaflopS Thanks for the feedback. Building of the test codes has manual steps - maybe I missed something in the build setup for the TEKKO_F3 test code. I've looked at the stall protection, and I think it is hard to find a compromise that both has high resilience to prop touching stuff and at the same time good protection. So it may end up as a programmable setting.

[dabit20]

The 32.62 test code still has some issues with spinning out of control on minor clipping of branches or touching the ground.

Also, yesterday I blew another Aikon AK32 6S ESC on a minor crash. Same symptoms as before: FET's are OK, 3.3V supply drops to ~1.6V, STM32F051 microcontroller runs very hot and obviously it does not report itself anymore in the BLHeli suite.

This time I had 3x 470uF low-ESR caps (EPCOS B41888C5477M000, ESR=64mOhm) on the power pads of 3 ESCs and the DShot wiring is twisted pair from the ESC to the FC and reduced rampup power to 20%.

I have a gut feeling that this 'killing more ESCs than props' started with the move to RPM filters and the later (>32.65) beta firmwares, but I have absolutely no data to back that up. I used to just cook motor windings and blow FETs instead of killing microcontrollers.

[dabit20]

Here is a video with the very quick acting protection of the ESC's, 32.62 test code: https://www.youtube.com/watch?v=gX2lLDi9buQ

As you can see, as soon as a prop touches something (in the video it is the prop touching the ground) the ESC stops spinning the motor entirely. This also happens when hitting a ghost branch (that otherwise would not cause much harm) and results in a spin towards the ground.

[sskaug]

What happens in that video seems to be that the ESC loses its input signal. You can hear the two beeps, the first for detecting signal (again) and the second for seing zero signal. I can not say why this happens - signal has to be gone for ~300ms before it is considered lost.

[dbpprt]

I'm experiencing exactly the same as @dabit20 but mid flight most of the time. 7" build with 2507 1500kv on 6s with Hobbywing X-Rotor ESCs. Sometimes I'm able to rearm mid-air, the sound sequence is the same as on the video above (had it on a GoPro video). I tried most of the test versions and all of them have this issue. Went back to 67 without any issues.

[dabit20]

Had another mid-flight failure today, turning the quad into a giant mudball. Just one in maybe 15 packs or so, but I am getting nervous punching the quad over high trees.

@sskaug: 300ms is quite some time, that is not what is happening in the video. Also, you see it happening when the prop is temporarily blocked. In the video it happens to be the front-left prop, but it might as well have been one of the others.

Taking clear pictures is quite impossible since it is all stuffed in little space and wrapped in tape, but here is a textual description of the drivetrain wiring: Power distribution is based on a piece of bare copper PCB sized 30x40mm or so with a slot milled in the middle creating two copper rectangles measuring approximately 40x15mm. 5S battery power is applied with 12AWG or 14AWG wires to the middle of those strips. XT60 was already replaced to rule out power issues, but as you can see in the first blackbox log plot the voltage seems no issue.

Separate ESC's are mounted on the arms, power to the ESC's is taken from ends of the strips and routed with a few cm of as parallel as possible (lowering inductance..) 18AWG conductors. Where enough arm real estate was available a 470uF/63mOhm ESR capacitor was mounted on the ESC power pads with a total of 3 pcs. Thus, one ESC has no extra capacitor.

Control wiring consists of two twisted wires running from the ESC directly to the pads of the (Foxeer F722 Dual) FC. Since GND is connected both on the ESC side and FC side I may have a ground loop causing issues. Not too likely (the setup used to perform fine before I started messing with RPM filters and beta firmware, I also have a very low DShot packet error rate even with DShot1200), but I might try to change the control wiring to small pieces of 1.13mm coaxial cable with the shield only connected at the ESC side.

All in all I do not think that the drivetrain wiring is causing the issues, but I might try to change back to a non-RPM setup using the regular stable firmware and I might try the small coax cable routing without ground connection at the FC side to rule that out.

[sskaug]

Thanks for the feedback. There is one potential change in 32.6.1 that could cause this, maybe in combination with how BF handles bidirectional dshot. We have proposed to BF to change to inverted dshot for bidirectional, hopefully we can then make a more robust system overall.

[sskaug]

I have now posted a new test code 32.6.2, where the stall protection is weak again. Going forward we will probably make this programmable, with good stall protection being default.

[dabit20]

I saw this on the Betaflight slack. I will try it and see if it works better for me than 32.67 Joe Lucid also has inverted bidir dshot code running, I will try that too.

[dabit20]

Is that same weak stall protection in 32.63? Flew that with the inverted bidir DShot today and tested whether it was possible to persuade 32.67 to uncommanded movement with the inverted BF code. So far it has been solid, no issues at all. And I did crash and tried to fly through trees. BF rescue mode kicks in for a split second and I can happily continue flight if I wish to do so, just like I used to be able to do.

It is not a scientific test since I also made another change: replaced the twisted pair signal wiring from FC to ESCs with coaxial cable, GND only connected at the ESC side to prevent ground loops and stray currents through the signal grounds. I chose the ESC side for practical reasons only.

[sskaug]

Sounds good, then. Protection is the same for 32.6.2 and 32.6.3.

[TheZeroBeast]

Hi @sskaug,

I think I may be seeing this issue as well, I've tried 32.6 and 32.6.2 located here

Running this on the 4-in-1 J Bardwell ESC with an F10 Strix.

But still get random flip outs that look very similar to the above black box graph. Mine from today on 32.6.2 is attached.

Different from default setup I've made the following changes if that helps get to the bottom of this.

• 48kHz PWM frequency
• Autotiming
• 20% ramp up
• Sine wave
• High demag comp

[ExaflopS]

@TheZeroBeast What are You trying to test? I think beta blheli FW is actually aimed to use with bidir dshot and betaflight RPM filters, which works only with betaflight 4.0 and higher. If You use strix F10, I would reccomend to use stable blheli fw. Also, there are known cases where F10 itself or fw could have problems causing flipouts.

[TheZeroBeast]

Hey @ExaflopS,

Thanks for the speedy reply.

I’m not testing anything. I am trying to get my build stable as I have been plagued with this issue even on stable BLHeli_32.

I was unaware of those strix flip out issues and think that’s the nail for it. I have a CLRacing F7 going in tomorrow, hopefully fixing this so I’m able to go forth and test bidir DSHOT with RPM filtering.

[sskaug]

New test codes 32.6.4 and 32.6.5 are now published. Stall protection is made programmable, and the default mode is "normal", which gives more protection than 32.6. Setting it to "relaxed" gives similar behavior as 32.6.

[ExaflopS]

@sskaug Does it mean, Setting protection to "relaxed" gives less protection than "normal"?

[sskaug]

The new code has more protection by default (normal). Relaxed protection is less protection than normal.

[cosmofpv]

I got kind of the same issue with a brand new aikon 35A 20x20 ESC. My blackbox logs look exactly like the one form the tread opener. Every time i force the quad into propwash i get the issue. I tryed both stall protection options as well as nearly every other blheli32 option. But nothing helps. The FC sends the full throttle command to a motor (they are all 4 infected depending on the type of situation) but the esc is not executing them. The quad kind of tumbles for like 0.25 sec and then recovers itself. Same happens if i hit a branch, so i guess it happens every time the esc thinks the motor stops. Is there something else then the stall protection that i can try ? Or maybe some version without stall protection at all ?