search

bird-sanctuary / bluejay

:bird: Digital ESC firmware for controlling brushless motors in multirotors
GNU General Public License v3.0
301 stars 32 forks source link

2 failed ESCs on bluejay 0.20.0 #146

Closed jli closed 6 months ago

jli commented 1 year ago

Describe the issue

tl;dr I seem to have fried 2 new ESCs within ~1 hour of unremarkable flying, and the only thing that seems unique about my setups is that both were using Bluejay 0.20.0, so perhaps there's a bug...?

This may be total coincidence, but I built 2 new tiny whoops using the HappyModel ELRS F4 5A AIO flight controller, and both whoops seem to have bad ESCs within just ~10 packs/~1 hour of flying. I found this surprising because they both failed very quickly without any particularly nasty crashes. I acknowledge this may be total coincidence, or some other issue (maybe both FCs came from the same bad batch..?), but one more unusual thing about both of these builds is that I was using Bluejay 0.20.0, which I see was just released recently and seems to include some substantial changes/refactoring. So, I'm filing this just in case there is some bug that's causing hardware damage.

Some additional details:

Bluejay version

0.20.0

ESC variant

O_H_5

PWM frequency

48

DShot bitrate

300

Bidirectional DShot

On

FC firmware

Betaflight 4.4.2

Motor size

802, 702

Configurator debug log

No response

damosvil commented 1 year ago

Please, could you upload here a diff all and a photo of the builds?

stylesuxx commented 1 year ago

That's the same AIO that's being used in the Mobula6. I have flown dozens of packs with it during development without issues.

Hot motors are usually a sign of an imperfect tune. This does not explain why the motor got hot when bench testing, since PID loop is not interfering with the motor tab.

Can you visually see any damage on the FETs?

jawz101 commented 1 year ago

Happymodel Mobula 7- after updating to v0.20.0 it eventually stopped arming where motors won't spin. I reverted to 0.19 and then back to blheli_s and now none of the firmwares won't spin motors. On the config page I did make sure the 1s vs 2s+ setting was set to 1s since the FC only supports 1s.

Coincidental that I came out here to look at the repo to see how the 0.20. build was working because I have a Mobula 7 as well and it's now no longer working at all.

stylesuxx commented 1 year ago

none of the firmwares won't spin motors.

So the motors spin or not?

Please be aware that the Mobula 7 originally comes flashed with BLHeli_M, so there actually is no "back to BLHeli_S".

PS.: In the future please open your own issue, otherwise it is really hard to keep track of the issues and users.

jawz101 commented 1 year ago

I will post a separate issue

stylesuxx commented 1 year ago

Hey @jli - any update on this?

jli commented 1 year ago

Thanks @stylesuxx and @damosvil for the quick response, apologies for the delay.

Please, could you upload here a diff all and a photo of the builds?

Here are photos of the top and bottom of the 2 boards. The first board has a dead ESC for motor 1, and the second board has a dead ESC for motor 2. (Note that for my builds, I rotated the boards 90° CW and remapped the motors. I've marked the location of the dead ESCs in the photos to make it clear.)

photos of board with dead ESC 1:
![deadesc1_top](https://github.com/bird-sanctuary/bluejay/assets/133466/b3035406-5d82-4418-ad29-175bec5efa15) ![deadesc1_bottom](https://github.com/bird-sanctuary/bluejay/assets/133466/16f961c2-fa16-47ca-a63a-db57208e365c)
photos of board with dead ESC 2:
![deadesc2_top](https://github.com/bird-sanctuary/bluejay/assets/133466/83f15759-7eef-43ae-b898-ccd11147bb1d) ![deadesc2_bottom](https://github.com/bird-sanctuary/bluejay/assets/133466/ed6e93ca-8581-4eb8-85cb-ba62b842163d)

Here are the configs.

That's the same AIO that's being used in the Mobula6. I have flown dozens of packs with it during development without issues. [...] Can you visually see any damage on the FETs?

That's fair, I know it's a popular board. I'm not exactly sure what FET damage looks like, but I don't think I see any damage. All of the FETs on each of the boards look the same to me.

stylesuxx commented 1 year ago

Usually the broken FETs are easy to spot since they "pop", I circled the ones that look broken to me, they don't match your arrow though... 256840213-b3035406-5d82-4418-ad29-175bec5efa15

Profile 0 seems rather aggressive. I am personally not a fan of just using presets for tunes. Just because something says 1S or 65mm it does not fit everything. The diffs are very similar, so I am not overly surprised that the FETs went...

Software won't fix your issues at least with the first one where I am very confident to see popped FETs.

@damosvil asked for build pics in order to see the other components used - would you mind sharing which motors and props you used, just so that we can put the tune into a better context?

jli commented 1 year ago

Usually the broken FETs are easy to spot since they "pop", I circled the ones that look broken to me, they don't match your arrow though...

Yeah, those are the FETs by the plug for M3... I think that was some kind of residue on the FETs. I cleaned them off (just scraping with my fingernail), and it seems that got rid of it. Would a popped FET be able to be "cleaned up" that way?

dead_esc1_cleaned

Profile 0 seems rather aggressive. I am personally not a fan of just using presets for tunes. Just because something says 1S or 65mm it does not fit everything. The diffs are very similar, so I am not overly surprised that the FETs went...

Ah, yeah... I believe that profile is from the UAV-Tech 1-2S whoop preset. I don't recall if I was using that profile, or the more relaxed profile 1 or 2, but it's certainly possible I was using profile 0. Could you elaborate on what part of the tune is aggressive? The PID values, or other things like filters? (My usual tune is BF defaults + RPM filtering + dynamic idle, with D slider bumped up to 1.3. I think I'll go back to that..)

@damosvil asked for build pics in order to see the other components used - would you mind sharing which motors and props you used, just so that we can put the tune into a better context?

Yes sorry, I had scavenged the parts from those builds. I do have a photo of the build with dead ESC 1 though: mob6_memecam 65mm build, motors are happymodel 702 26000kv, props are gemfan 31mm biblades. I think dry weight was ~20g.

I'm not as sure about the 2nd build, but I believe it was a 75mm build, with happymodel ex802 22000 or 25000kv motors, and gemfan 40mm biblades.

stylesuxx commented 1 year ago

Yeah, those are the FETs by the plug for M3... I think that was some kind of residue on the FETs. I cleaned them off (just scraping with my fingernail), and it seems that got rid of it. Would a popped FET be able to be "cleaned up" that way?

No, if it's popped, it's really mechanically damaged, you can't clean that - I'll see if I can find something in my parts bin as reference. Your clean version looks fine.

Your build seems sane, no crazy high KV motors, frame looks fresh, not over propped.

I assume the stuttering of the motor does not go away once you give more throttle, right? Im am just asking because 65mm 1S builds often like min/max startup power to be cranked up all the way or they might not start up smoothly on arm, but since it was once working, this is a long shot and probably won't change anything.

My usual tune is BF defaults + RPM filtering + dynamic idle, with D slider bumped up to 1.3. I think I'll go back to that...

Yes, I would highly recommend you to do that. If you change filtering or disable filters completely you should do short flights and check if the motors get overly warm, that is usually a good indicator that something is not right. In the best case you chose a FC with Blackbox so you have proper data to see what is happening.

This is a bit OT and heavily opinionated: I don't trust those AIO boards with built in VTX. They have all failed on me very, very quickly. Everything on the board is generating heat, adding a VTX which adds more heat is in my opinion really operating the hardware at it's maximum. I don't think that this was the culprit here, just something you might want to consider in the future. I like to go with an AIO that has a Blackbox instead and go with a separate VTX...

jli commented 1 year ago

I assume the stuttering of the motor does not go away once you give more throttle, right? Im am just asking because 65mm 1S builds often like min/max startup power to be cranked up all the way or they might not start up smoothly on arm, but since it was once working, this is a long shot and probably won't change anything.

No it doesn't go away. I would ramp up quickly from 0 to eg 1500 and the stuttering behavior was the same as ramping up slowly to values like 1100.

This is a bit OT and heavily opinionated: I don't trust those AIO boards with built in VTX. They have all failed on me very, very quickly. Everything on the board is generating heat, adding a VTX which adds more heat is in my opinion really operating the hardware at it's maximum. I don't think that this was the culprit here, just something you might want to consider in the future. I like to go with an AIO that has a Blackbox instead and go with a separate VTX...

Interesting, I've been doing more builds lately with separate VTX. Originally I was just motivated by having to pay less to replace broken parts, but it makes sense that more heat on a small board results in less reliability. Not to go even more OT, but I'm curious if you have particular components to recommend? I've used the happymodel cross f4 and the betafpv f4 (also cross shaped) for FCs and the happymodel ovx300 as the VTX. I'm currently evaluating other options for future builds, hoping to optimize for reliability 😅

stylesuxx commented 1 year ago

No it doesn't go away. I would ramp up quickly from 0 to eg 1500 and the stuttering behavior was the same as ramping up slowly to values like 1100.

Yeah, then the ESC is definitely gone. If you are adventurous you could try transplanting a set of good FETs from one board to the other.

I'm curious if you have particular components to recommend?

It's really difficult, I think everyone has their go to manufacturers. Personally I have had the best experiences with TuneRC - they have a cross FC with BB, but without RX and VTX, they might soon come out with the same one but including an ELRS RX. Other than that I have had good luck with the BetaFPV AIO cross that also comes with BB. Please take this with a grain of salt - I am not giving you a guarantee that it will work for you, but they have worked well for me and the Support is responsive.

jli commented 1 year ago

Yeah, then the ESC is definitely gone. If you are adventurous you could try transplanting a set of good FETs from one board to the other.

Yep, I'd like to learn how to do these types of repairs so I can give it a shot. Is it pretty much always the FETs? Is it surprising/odd that there's no visible damage to mine?

Thanks for all your help @stylesuxx 🙏

schome1 commented 12 months ago

Although I can't confirm that v0.20 burned up three of my AIO FC ESCs, I can confirm that v0.20 was producing severe bounce-back during high-speed turns. I switched back to Bluejay v0.19 and the problem was gone. I've confirmed this on 8 TuneRC Poly F405 flight controllers. The two that used Bluejay v0.19 had no bounce-back issue. The others all had bounce-back issues during high speed turns while rolling left, and increasing throttle. All of the 8 builds had exactly the same hardware and electronics setup, using the same frames and the same props.

The three TuneRC Poly F405 flight controllers that burned up, all had Bluejay v0.20 installed, but I cannot confirm whether that was caused by the firmware or if it was because on those three flight controllers, I chose not to install a capacitor.

My drones were all configured with the following:

I hope this information helps.

damosvil commented 12 months ago

@schome1 please, could you provide blackbox data using 0.19 and 0.20? We also would need captures of esc-configurator.com settings, a diff all of the Betaflight configuration and a photo of the build to have more context about the conditions in which the issue happens.

schome1 commented 12 months ago

@damosvil I will work on gathering that information this weekend.

schome1 commented 12 months ago

requested-files.zip The requested files have been uploaded. I was unable to reproduce the severe bounce-back that I had described, but while flying with Bluejay v0.20, I fried my fourth TuneRC Poly F405. The pattern is that all four have fried immediately after attempting a crash-flip. I don't hold the sticks down when I do a crash-flip. There were only 33 flights on it when it burned up. There were 17 flights on one of them, there were 10 on another, and 1 on another. I have other TuneRC Poly F405 flight controllers that have over 1000 flights combined, through many crash-flip modes, many races, many crashes. Maybe it's not v0.20, but that seems to be the common link between all of the burned flight controllers.

I hope you find the files useful. Please let me know if I can get any more data for you.

damosvil commented 11 months ago

@schome1 I have been checking the blackbox data, and I think that there is a reason for blowing the fets.

Version 0.19 updates extra telemetry data very often, every 16ms, so it is more prone to lose zero crossings and cut power to engage next zero crossing. For this reason 0.19 it is more forgiving on heating when handling agressive tunnings.

On the other hand version 0.20 updates extra telemetry data every 128ms, so it is much more robust on detecting zero crosings but if the tune is agressive it will try to do what it is asked to do, i.e. to update motor power as fast as possible. This causes motor heating, esc heating, and fet stress on agressive tunnings.

In addition to that what happens during crashes is that sometimes motor gets stuck, and esc desyncs. Unfortunately when this happens the fc tries to compensate increasing power on the motor to avoid tilting. If esc is not disarmed fast it heats and fets stresses.

Blheli_S and Bluejay up to 0.20 had a bug that caused power not to be limited during startup phases. So in the case the esc is not disarmed in a crash the fc may be sending high throttle to the esc, with a motor stuck and full power. So here we have the perfect storm, and the probability of killing escs is very high.

In a few days we will release 0.20.1, that correctly applies max startup power during motor startup. That will help on the crashes scenario. But you need to rebuild your tune.

Going back to blackbox data what I see is mainly fc is sending commands to the motors to correct gyro noise. This is normally caused by bad filtering and overagressive tunes.

I encourage you to:

This is the motor handling in your tunes: imagen

This is the motor handling in a good tune: imagen

jli commented 11 months ago

@damosvil Thanks for all those details!

Blheli_S and Bluejay up to 0.20 had a bug that caused power not to be limited during startup phases. So in the case the esc is not disarmed in a crash the fc may be sending high throttle to the esc, with a motor stuck and full power. So here we have the perfect storm, and the probability of killing escs is very high.

This is my understanding, could you confirm is this is correct?

On the other hand version 0.20 updates extra telemetry data every 128ms, so it is much more robust on detecting zero crosings but if the tune is agressive it will try to do what it is asked to do, i.e. to update motor power as fast as possible. This causes motor heating, esc heating, and fet stress on agressive tunnings.

Just curious, what's the extra telemetry data? Is that like ESC temp?

damosvil commented 11 months ago

This is my understanding, could you confirm is this is correct?

* When arming for the first time, "max startup power" limits the amount of initial throttle command sent by the ESC to get the motor started. This helps protect the FETs from having too much power that can't be dissipated, in case there's something wrong (motor unable to spin for whatever reason).

When arming for the first time "max startup power" DO NOT limit the amount of initial throttle command sent to the ESC to get the motor started. ESC is NOT protected. There is a bug that causes this. Fixed at: https://github.com/bird-sanctuary/bluejay/pull/156

* Whenever the ESC desyncs / the motor stops spinning, this is a "startup phase" since the ESC needs to start spinning the motor again.

Correct.

* Blheli and Bluejay both do _not_ apply the "max startup power" setting in this situation, so it may send a high throttle value and a lot of current through the FETs even if the motor is stuck. This affects Bluejay 0.20.0 also, but 0.20.1 will have a fix for this.

Correct. Not in this situation, neither when arming for the same time.

Just curious, what's the extra telemetry data? Is that like ESC temp?

https://github.com/bird-sanctuary/extended-dshot-telemetry

For now Bluejay sends relevant data in temperature frame, stress level frame and status frame.

jli commented 11 months ago

When arming for the first time "max startup power" DO NOT limit the amount of initial throttle command sent to the ESC to get the motor started. ESC is NOT protected. There is a bug that causes this. Fixed at: https://github.com/bird-sanctuary/bluejay/pull/156

Hm, if max startup power wasn't limiting power on first arm or during desyncs, was it just never active?

damosvil commented 11 months ago

Hm, if max startup power wasn't limiting power on first arm or during desyncs, was it just never active?

https://github.com/bitdump/BLHeli/blob/20c36c2695c5b5ea855288e0046313688d1367a5/BLHeli_S%20SiLabs/BLHeli_S.asm#L1683

imagen

imagen

damosvil commented 11 months ago

@schome1 Please, could you try this filter settings and send me back a blackbox?

imagen

Also please, try reducing damping to 1.2 in pid tab.

nuuls commented 11 months ago

Hi! Does this also affect the power limit when crashing mid-flight? I've had it happen a few times where I bend a motor bell, and the motor is already smoked by the time I can disarm.

stylesuxx commented 11 months ago

Hi! Does this also affect the power limit when crashing mid-flight? I've had it happen a few times where I bend a motor bell, and the motor is already smoked by the time I can disarm.

Yes - also see here: https://github.com/bird-sanctuary/bluejay/issues/161

ellmotion commented 11 months ago

I also had an issue with a BetaFPV F4 1S 12A AIO FC V3 (Pavo Pico) after upgrading to 0.20. I had a crash where the drone ended up upside down. I tried to use turtle mode to flip it back over and saw smoke in my goggles. The FC was destroyed by the flames. I have no idea if this is a coincidence following the upgrade to 0.20. IMG_1258 Unfortunately, I am unable to retrieve any information from it because the USB interface has also been burned.

stylesuxx commented 11 months ago

The max startup fix in v0.20.1 should help in preventing such issues in the future. Turteling is a stress test for all involved components, if motors are blocked this can quickly lead to bad situations, especially when trying to push too far, too quick. Logs would have been great thou.

Sorry for your loss in any case...

schome1 commented 11 months ago

@schome1 Please, could you try this filter settings and send me back a blackbox?

imagen

Also please, try reducing damping to 1.2 in pid tab.

Still with v0.20? I have been flying v0.19 with no issues this week. I have 108 flights in on this build. In my previous four builds (v0.20) I had a combined sum of 60 flights and all of them burned up.

Let me know if you want me to use v0.20 or if an update is available to try (I didn't see one when I checked). I'm going to wait to test with these settings until another flight controller arrives because I don't want to be left with none in the event that this one burns up too. It looks like it could arrive tomorrow, so it's very possible that I could test tomorrow or Sunday.

damosvil commented 11 months ago

Yes, with 0.20. The test would consist of trying those settings to check that if motor handling improved, but no need to crash, neither using turtle mode. Only 10 seconds of flight time to extract a blackbox and knowing if improving filtering and using less agressive pid settings are good for you.

schome1 commented 11 months ago

bluejay-v0.20-pid-filter-test.bbl.zip Here is the blackbox log you requested. I did change my frame - same size, same exact dimensions, just switched to a CF frame instead of G11. It's a little lighter. I did change my tune as well, and it's been flying great. I think the vibrations that were seen in the previous log files could have been from well used props at the time the log was saved.

This log is a short flight with Bluejay v0.20 and the PIDs/Filters as you suggested, plus new props.

damosvil commented 11 months ago

@schome1 many thanks for the log. We are double checking 0.20 code to improve startup protection. For now please stick to 0.19.2

vad7 commented 11 months ago

CrazyF405HD after updating to 0.20 burned in flight:

20230823_095431

damosvil commented 11 months ago

@vad7 If you are still be able to connect to the AIO, please, could you extract the blackbox data and upload it? Could you also let us know what motors, frame are you using and upload some photos of the build?

hyp0dermik-code commented 11 months ago

I see @schome1 was running dynamic idle @ellmotion @vad7 did you have dynamic idle in BF enabled? A config dump or any log would tell us if you have one

vad7 commented 11 months ago

yes. Happymodel Mobula8 stock (EX1103 KV11000 9N12P) BTFL_cli_backup_Mobula8_20230820_202604.txt

ellmotion commented 11 months ago

yes, dynamic idle was enable, Unfortunately, the fc is out of service and I have no dump. it was a pavo pico stock. FC: F4 1S 12A AIO FC V3 Motors : BETAFPV 1102 14000kv

jli commented 11 months ago

just fyi for folks who've chimed in on this issue, it looks like 0.20.0 got pulled for potentially damaging ESCs/motors: https://github.com/bird-sanctuary/bluejay/releases/tag/v0.20.0

Screenshot_20230830-091711.png

k8ieone commented 11 months ago

From what I'm reading here, that seems like a good move. I almost switched from BLHeli_m, but I happened to check open issues first. I'll wait until this issue is sorted out before I switch.

damosvil commented 11 months ago

https://youtu.be/RTcNqdShRx8?feature=shared @vad7 I found the video of the flight. Was the kwad a custom build? Do you have any photo of the kwad before the incident? I saw that when it took off core temp was 53 deg C. It suddenly raised over 60 deg C the last 3 seconds of the flight

It seems that the vtx was radiating heat over the mosfets, and resistor networks. In that AIO the F405 MCU is in the opposite side of the vtx, and it was sensing 53 deg C. As I said mosfets and resistor networks are located in the side besides the vtx, so they were hotter for sure. imagen

vad7 commented 11 months ago

@damosvil Mobula8 Walksnail stock. Walksnail Avatar HD Mini 1S Lite is very hot but cools down quickly during flight. There were about 50 flights on firmware 0.19, the second flight on 0.20 ended like this. Walksnail survived.

stylesuxx commented 10 months ago

I would like to encourage everyone in this thread to try our new RC of v0.20.1 - can be installed via esc-configurator.com and should fix all motor/esc protection issues

hyp0dermik-code commented 10 months ago

I would like to encourage everyone in this thread to try our new RC of v0.20.1 - can be installed via esc-configurator.com and should fix all motor/esc protection issues

When the startup power settings have been configured correctly by following https://github.com/bird-sanctuary/bluejay/wiki/Setup#startup-power-and-motor-idle-setup

stylesuxx commented 6 months ago

I would like to encourage everyone in this thread to try our new RC of v0.21.0 - can be installed via esc-configurator.com

Closing this issue down for the time being.