Roll of death with Bidirectional DSHOT on H7EXTREME

[joelucid]

Describe the bug Role of death with activated Bidirectional DSHOT and RPM Filter on SPRACINGH7EXTREME

To Reproduce Bidirectional DSHOT was fixed with this issue #8836. I updated to Build #1900 (28.09.2019 22:50:24) and activated rpm filter. Everything else is set to default. Sometimes it happens that the quad performs a role of death. (Log is attached LOG00008.zip)

Without activated bidir and rpm filter it works great with no issues.

Flight controller configuration

Betaflight / SPRACINGH7EXTREME (SP7E) 4.1.0 Sep 28 2019 / 23:25:54 (fcaa16cd7) MSP API: 1.42

start the command batch

batch start

board_name SPRACINGH7EXTREME

name: Marmotte

resources

resource MOTOR 1 A03 resource MOTOR 2 A02 resource MOTOR 3 A01 resource MOTOR 4 A00

feature

feature -TRANSPONDER

beeper

beeper -GYRO_CALIBRATED beeper -RX_LOST_LANDING beeper -DISARMING beeper -ARMING beeper -ARMING_GPS_FIX beeper -BAT_CRIT_LOW beeper -BAT_LOW beeper -GPS_STATUS beeper -ACC_CALIBRATION beeper -ACC_CALIBRATION_FAIL beeper -READY_BEEP beeper -DISARM_REPEAT beeper -ARMED beeper -SYSTEM_INIT beeper -ON_USB beeper -BLACKBOX_ERASE beeper -CRASH_FLIP beeper -CAM_CONNECTION_OPEN beeper -CAM_CONNECTION_CLOSE beeper -RC_SMOOTHING_INIT_FAIL

serial

serial 0 64 115200 57600 0 115200 serial 1 32 115200 57600 0 115200 serial 4 2048 115200 57600 0 115200

led

led 0 11,11::C:2 led 1 12,12::C:2 led 2 13,13::C:2 led 3 14,14::C:2 led 4 15,15::C:2 led 5 5,15::C:2 led 6 6,15::C:2 led 7 7,15::C:2 led 8 8,15::C:2 led 9 9,15::C:2 led 10 4,11::C:2 led 11 3,12::C:2 led 12 2,13::C:2 led 13 1,14::C:2 led 14 0,15::C:2

aux

aux 0 0 0 900 1300 0 0 aux 1 13 2 1700 2100 0 0 aux 2 15 6 900 1075 0 0 aux 3 35 4 900 1300 0 0

adjrange

adjrange 1 0 6 1100 2100 30 6 0 0

rxrange

rxrange 0 987 2011 rxrange 1 990 2011 rxrange 2 987 2011 rxrange 3 990 2011

vtxtable

vtxtable bands 5 vtxtable channels 8 vtxtable band 1 BOSCAM_A A CUSTOM 5865 5845 5825 5805 5785 5765 5745 5725 vtxtable band 2 BOSCAM_B B CUSTOM 5733 5752 5771 5790 5809 5828 5847 5866 vtxtable band 3 BOSCAM_E E CUSTOM 5705 5685 5665 5645 5885 5905 5925 5945 vtxtable band 4 FATSHARK F CUSTOM 5740 5760 5780 5800 5820 5840 5860 5880 vtxtable band 5 RACEBAND R CUSTOM 5658 5695 5732 5769 5806 5843 5880 5917 vtxtable powerlevels 3 vtxtable powervalues 13 20 26 vtxtable powerlabels 25 100 400

master

set dyn_notch_width_percent = 0 set dyn_notch_q = 250 set mag_hardware = NONE set min_check = 1010 set max_check = 1990 set serialrx_provider = SBUS set blackbox_p_ratio = 64 set dshot_bidir = ON set motor_pwm_protocol = DSHOT600 set align_board_yaw = 180 set vbat_min_cell_voltage = 350 set vbat_warning_cell_voltage = 360 set yaw_motors_reversed = ON set small_angle = 180 set deadband = 3 set yaw_deadband = 3 set pid_process_denom = 1 set ledstrip_race_color = GREEN set osd_vbat_pos = 384 set osd_rssi_pos = 2106 set osd_tim_2_pos = 2423 set osd_current_pos = 406 set osd_mah_drawn_pos = 2456 set osd_altitude_pos = 2401 set osd_warnings_pos = 14730 set osd_avg_cell_voltage_pos = 2433 set osd_disarmed_pos = 2347 set osd_stat_max_spd = OFF set osd_stat_endbatt = ON set osd_stat_battery = ON set osd_stat_bbox = OFF set osd_stat_max_fft = ON set debug_mode = RPM_FILTER set vtx_band = 5 set vtx_channel = 8 set vtx_power = 3 set vtx_low_power_disarm = UNTIL_FIRST_ARM set vtx_freq = 5917 set vcd_video_system = NTSC set name = Marmotte

profile 0

profile 0

set vbat_pid_gain = ON set feedforward_transition = 30 set iterm_relax_type = GYRO set iterm_relax_cutoff = 10

rateprofile 0

rateprofile 0

set roll_rc_rate = 95 set pitch_rc_rate = 95 set yaw_rc_rate = 105 set roll_expo = 30 set pitch_expo = 30 set yaw_expo = 20 set roll_srate = 82 set pitch_srate = 82 set yaw_srate = 82

end the command batch

batch end resource show all Currently active IO resource assignments: (reboot to update)

A00: MOTOR 4 A01: MOTOR 3 A02: MOTOR 2 A03: MOTOR 1 A04: FREE A05: FREE A06: VTX_DATA 10 A07: VTX_CLK 10 A08: LED_STRIP A09: FREE A10: FREE A11: USB A12: USB A13: FREE A14: FREE A15: GYRO_CS 1 B00: VTX_CS 10 B01: VTX_POWER B02: QSPI_CLK 1 B03: SPI_SCK 3 B04: SPI_MISO 3 B05: FREE B06: FREE B07: FREE B08: I2C_SCL 1 B09: I2C_SDA 1 B10: QSPI_BK1CS 1 B11: FREE B12: GYRO_CS 2 B13: SERIAL_TX 5 B14: FREE B15: SERIAL_RX 1 C00: ADC_CURR C01: ADC_BATT C02: SPI_MISO 2 C03: SPI_MOSI 2 C04: ADC_RSSI C05: ADC_EXT C06: FREE C07: FREE C08: SDIO_D0 C09: SDIO_D1 C10: SDIO_D2 C11: SDIO_D3 C12: SDIO_CK C13: FREE C14: FREE C15: FREE D00: FREE D01: FREE D02: SDIO_CMD D03: SPI_SCK 2 D04: GYRO_EXTI D05: SERIAL_TX 2 D06: SPI_MOSI 3 D07: BEEPER D08: FREE D09: FREE D10: SDCARD_DETECT D11: QSPI_BK1IO0 1 D12: QSPI_BK1IO1 1 D13: QSPI_BK1IO3 1 D14: FREE D15: FREE E00: FREE E01: FREE E02: QSPI_BK1IO2 1 E03: LED 1 E04: SYSTEM E05: CAMERA_CONTROL E06: FREE E07: QSPI_BK2IO0 1 E08: QSPI_BK2IO1 1 E09: QSPI_BK2IO2 1 E10: QSPI_BK2IO3 1 E11: OSD_CS E12: SPI_SCK 4 E13: SPI_MISO 4 E14: SPI_MOSI 4 E15: GYRO_EXTI F00: FREE F01: FREE F02: FREE F03: FREE F04: FREE F05: FREE F06: FREE F07: FREE F08: FREE F09: FREE F10: FREE F11: FREE F12: FREE F13: FREE F14: FREE F15: FREE G00: FREE G01: FREE G02: FREE G03: FREE G04: FREE G05: FREE G06: FREE G07: FREE G08: FREE G09: FREE G10: FREE G11: FREE G12: FREE G13: FREE G14: FREE G15: FREE

Currently active Timers:

TIM1: CH1: LED_STRIP TIM2: FREE TIM3: FREE TIM4: FREE TIM5: CH1: MOTOR 4 CH2: MOTOR 3 CH3: MOTOR 2 CH4: MOTOR 1 TIM6: FREE TIM7: FREE TIM8: FREE TIM12: FREE TIM13: FREE TIM14: FREE TIM15: CH1: CAMERA_CONTROL TIM16: FREE TIM17: FREE

Currently active DMA:

DMA1 Stream 0: MOTOR 4 DMA1 Stream 1: MOTOR 3 DMA1 Stream 2: MOTOR 2 DMA1 Stream 3: MOTOR 1 DMA1 Stream 4: FREE DMA1 Stream 5: FREE DMA1 Stream 6: FREE DMA1 Stream 7: FREE DMA2 Stream 0: ADC 1 DMA2 Stream 1: ADC 3 DMA2 Stream 2: LED_STRIP DMA2 Stream 3: FREE DMA2 Stream 4: FREE DMA2 Stream 5: FREE DMA2 Stream 6: FREE DMA2 Stream 7: FREE

dshot_telemetry_info

Dshot reads: 268399 Dshot invalid pkts: 466 Dshot directionChange cycles: 984, micros: 2

Motor eRPM RPM Hz Invalid ===== ======= ====== ===== ======= 0 0 0 0 0.66% 1 0 0 0 0.00% 2 0 0 0 0.00% 3 0 0 0 0.00%

117 133 149 165 197 213 229 246 278 294 310 326 342 358 406 14 0 0 3373138432 1990328054 572784642 4290238063 16 16 16 32 16 16 17 32 16 16 16 16 16 48 4294966904 4294967282 0 3373138432 2912156918 2877423884 3717453421

tasks Task list rate/hz max/us avg/us maxload avgload total/ms 00 - ( SYSTEM) 9 12 0 0.5% 0.0% 0 01 - ( SYSTEM) 999 14 1 1.8% 0.5% 124 02 - ( GYRO/PID) 8002 55 47 44.5% 38.1% 41653 03 - ( ACC) 999 26 9 3.0% 1.3% 1057 04 - ( ATTITUDE) 99 18 4 0.6% 0.5% 58 05 - ( RX) 32 34 20 0.6% 0.5% 81 06 - ( SERIAL) 99 314493 0 3113.9% 0.0% 665 07 - ( DISPATCH) 998 13 0 1.7% 0.0% 30 08 - (BATTERY_VOLTAGE) 49 14 1 0.5% 0.5% 6 09 - (BATTERY_CURRENT) 49 12 1 0.5% 0.5% 2 10 - ( BATTERY_ALERTS) 4 13 1 0.5% 0.5% 0 11 - ( BEEPER) 99 12 1 0.6% 0.5% 6 14 - ( BARO) 153 30 6 0.9% 0.5% 112 15 - ( ALTITUDE) 39 14 3 0.5% 0.5% 11 17 - ( TELEMETRY) 249 24 0 1.0% 0.0% 81 18 - ( LEDSTRIP) 99 34 3 0.8% 0.5% 40 20 - ( OSD) 59 234 13 1.8% 0.5% 104 22 - ( CMS) 59 11 1 0.5% 0.5% 3 23 - ( VTXCTRL) 4 25 0 0.5% 0.0% 2 24 - ( CAMCTRL) 4 12 0 0.5% 0.0% 0 26 - ( ADCINTERNAL) 1 12 1 0.5% 0.5% 0 27 - ( PINIOBOX) 19 12 0 0.5% 0.0% 1 RX Check Function 2 0 2 Total (excluding SERIAL) 62.3% 45.9%

Setup / Versions

Flight controller: H7EXTREME ESC: TekkoF3 Metall 65A with 32.7.0 of blheli32 Esc are connected via extra ground and pwn wires not twisted -F40 pro III 1600kv Additional context Here is a short video: https://youtu.be/7YiYsa2Esek

20191012_152817

[joelucid]

@maGfpv, @mateyhv, @nerocide, @haslinghuis, @benedikt-bartscher, @hydra:

I've just resigned from the Betaflight Members slack over the outrageous treatment of this issue. Unlike a few people on the Betaflight leadership team I do not feel that set dshot_bidir = off is in any way an acceptable workaround to your problem.

As creator of bidirectional dshot, rpm filtering and related features I'm committed to deliver a fix for you - even if it should have to be outside of Betaflight. I sincerely apologize to you for the behavior of my colleague.

[joelucid]

Original issue: https://github.com/betaflight/betaflight/issues/9098

[joelucid]

Do we have any new insights on the issue since it was locked down to prevent any external input?

[maGfpv]

hi and thank you !

Last thing i tried was the ESC calibration with multishot which was mentioned from @haslinghuis.

But this didnt change anything in my case. ESC start up was even more unsynced at startup without entering the cli. It also leads to role of death. Sadly i logged the wrong debug mode...

Just let me know how i can participate to get the issue solved. There are some guessings that compenents connected to the FC 5V line are making trouble. But hydra told me that "the capacitors for the 3v3 input and output are actually 4x over-specified for the regulators to make sure there's always power available and that it is clean as possible." And there is also no general schema which name a special component or so.

e.g..: "Another user reported on YouTube that Bidir/rpm filter is working for him with bf 4.1. on h7extreme Interesting, he had issues with motor signals on Motor 3 and 4 with errors above 50 % when he installed a buzzer to the board."

And also the issue from @emski0 with the cam connected to 5v...

[CertainBot]

Thanks for reopening the issue @joelucid. To be honest I was puzzled finding the original #9098 being marked as solved and closed with no further comment...

I tried the multishot calibration on a Mamba 506, (blheli32) 50A on 6s, although we were repeatedly being told that Dshot being a digital signal does not need a calibration. Seems that in the few tests I did there were few unsynchronized startups although they are still present. And there were a lot entering and exiting CLI. On four consecutive batteries (all good synced startups) the drone flew well but were very gentle flights since I had other non related issues. Did one hard push only, clearly not enough for testing the death roll possibility. Its hard for me to go harder since my actual flying spot is over a hard surface with too much water so cannot take the risk.

Its difficult to theorize without real data, but since its anly happening on the H750 board there must be some missunderstanding/misscomunication between this particular MCU and the ESC MCUs when Bidir is on.

I am not sure why there are so few inputs from hydra on this issue since its his board, its an expensive board, it has been a long time ago since most of the customers have bought it and expect to work flawlessly and last but not least I sent him a Mamba 506 almost a month ago for testing purposes as he aparently had no blheli32 based ESC around and we are still at the same point.

[joelucid]

Yeah - I'm sorry we seem to be having some difficulties atm. I was going to refer you to issue https://github.com/betaflight/betaflight/issues/9103 which also describes unsynchronized startup tones with blheli32. Yet this issue has been deleted overnight. 🤦‍♂

Be that as it may - I think we need to focus on the death roll issue. Have you tried the traditional death roll prevention techniques? Try with idle_min_hz=0 and dshot_idle_value fairly high - like 7% or so - way higher than you would usually set. If it still happens on multiple quads we can be fairly sure it's not a classical desync.

Please record a log with debug_mode=rpm_filter.

[joelucid]

I've been in contact with github. Looks like the user benedikt-bartscher who submitted 9103 has been flagged for some reason. It's done by algos - happened to me some time back, too, just for editing the rpm filter wiki page. Hopefully issue will be back soon.

[azixaka]

I have the same board (my second same board actually due to other issues) and I am thinking about putting it in my build with:

Aikon AK32 55A (has 5V 2A BEC which I could use to connect all the 5V consumers) T-Motor 2207 MCK 6S 1800kv motors X Bee Light MCK 2019 frame Matek Mini VTX with its own 5V 1A BEC 2x 1000uF Panasonic FS capacitors Foxeer Predator V4 Micro cam

And I have a brand new SP Racing H7 Extreme which I intend to try out because my Radix is so noisy and I can't fix it.

Any particular test case I could try for you to verify anything guys? Don't know maybe load the 5V line and see or the opposite - don't put anything on the 5V line and see etc?

[CertainBot]

Thanks @joelucid @haslinghuis. Issue 9103 still does not appear but since that is not a problem I won't bother. I cannot do any tests due to a long window of a horrible weather. Hope to do some tests once its over.

[haslinghuis]

About #9103 most interesting part (for me), I think this sums it up :

@sskaug is author or developer (or both) of BlHeli32 ESC firmware.

@sskaug said something like this: he thinks the out of sync esc initialisation is because some boot delay causing the ESC temporary entering bootloader mode and then timeouts and then does the full init.

Still note this behaviour is with bidir_dshot enabled and when using cold boot (plugging lipo in). On the bench with USB attached, when plugging LiPo the init is normal.

@sskaug does not have a H7 so he cannot reproduce since this is H7. On F7 there was a similar problem but that had to with double gyro detection. I tried the H7 with only gyro 1 enabled and with only gyro 2 enabled. No difference. So this is not the problem. Without gyro installed ESC will not initialize.

So, the big question is the boot sequence seems odd but seems not to influence flying performance? Or is it dangerous and causing desync?

From release 4.1 I never had motor errors (even on full throttle). This is without props! But using only 3S voltage and my max burst mode is about 4x22A + FC/VTX+Cam/RX/GPS. Purpose for this drone is long range, not racing nor freestyle. even do not use a low ESR capacitor on the power leads!

So waiting for other people with blackbox log files with debug_mode=rpm_filter because I cannot reproduce this other than the ESC out of sync init

@Mateyhv I cannot find the video but I saw Joshua Bardwell testing ESC/Motor problem on the bench with props on - the quad was mounted on a box on the table with elastics over the arms round the table. This could be dangerous.

[haslinghuis]

Imho, i doubt there is anything wrong with the H7 as I think @hydra has done a great job bringing the H7 but BF4.1 focus is on unified targets + rpmfilter and the code still has to adapt further to the changes in STM32H7 architecture..

[nerocide]

Thank you @joelucid for reopening this issue. Indeed closure of last one was quite brutal. Since then, i've dismantled the H7 unit and the kwad using it, I was not able to reproduce deathroll on my last tries. ESC were in sync. I'll see if it can get some data to crush soon.

[hydra]

@haslinghuis I doubt that boot delays would cause any issues in flying once the ESC has eventually detected the motor signals. Having ESC firmware that are tolerant of no-output signals on boot would be good.

I don't think there is a published specification, @joelucid or @sskaug can correct me here, of: a) maximum time from cold boot to first motor signal (valid frame) b) maximum 'dead' time, once motor signals have been supplied to the ESC. This happens e.g. during a reboot. If there are then this is something that can be checked? If not then this is something that should be published by ESC firmware authors right?

@Mateyhv / @haslinghuis If you're testing motors with props on, you can turn the props up-side-down so the quad sucks itself to the box/floor instead of trying to launch itself into your face. Take other safety precautions too, such as a net/cage/box over the whole quad when experimenting in this fashion.

[hydra]

@joelucid I added some scope traces to the startup tones issue that some people are experiencing.

https://github.com/bitdump/BLHeli/issues/410

Probably not related to any in-flight issues, but you and @sskaug would know better than I regarding that.

[hydra]

For those following, test ESC firmware in https://github.com/bitdump/BLHeli/issues/410 has shown to resolve the ESC initialisation tones. Odd/delayed initiation tones were simply the ESCs going into bootloader mode and then timing out, updated test ESC firmware resolved that.

[hydra]

Is anyone still having this issue? There have been no new reports from anyone for a long time now.

I recently ran into an issue on a prototype FC i'm working on where I'd get a death roll by bouncing it off the grass or stopping after a hard flip, the issue was fixed by adding a 470uF 50V cap to the 4in1 ESC I was using @Mateyhv 's Mamba 506 ESC that he kindly send me.

[maGfpv]

I swapped my h7 extreme. Because i was so frustrated about your board and that the issue is not fixed or nobody is trying to get it solved... I had roll of death and was running a 50v 1000uf low esr cap. So this wasnt a fix for me. Im running a kakute f7 with no issues now. Everything is working great. Rpm filter with no error. But sometimes i get unsynced ESC initialisation tones. But hey, this is ok it flies great. I will give your board another chance when its grown up and dont have any of theething troubles.

[hydra]

@maGfpv Ok, good to know you've tried adding a capacitor. Can you confirm that the other FC you installed used all the other components that were used with the H7EXTREME? It would be good if you have time to re-install it and create more logs, right now we've not got much to go on.

As noted above, the un-synced ESC tones are caused by the ESCs going into BlHeli bootloader mode and then the ESC bootloader timing out and then detecting a motor signal after the FC has booted, nothing to worry about with regards to flight. This could occur on any FC and isn't H7 specific. See https://github.com/bitdump/BLHeli/issues/410 for details on that.

[hydra]

@maGfpv Just a thought, but there's probably some new ESC firmware out for your ESCs since you last tried it, would be good to get before/after logs too.

[nerocide]

@hydra actually I stopped using the fc with rpmfilter as it's not stable setup. Will get a it running soon to see if there is changes

[maGfpv]

@hydra

I will try maybe this summer. But my build is flying perfect now. So I won't change anything...

Interesting to see that I'm not the only one who swapped his board to another one.

[DanO83]

Maybe I will give it a try again. The troubles begon when I swapped my Matek F722 for the H7 Extreme, nothing was changed except the FC. Now I swapped out the old motors, added an extra 470uf capacitor (I already had installed one) and updated to the latest BF 4.1.2. Just waiting for a change in the weather.

What logs you need me to make?

[hydra]

@danielolsman Hiya, yes that would be helpful. Just enable logging and try and fly it around and see if you can reliably get it to repeat the problem or not. Fly over something soft and not too high up or far away, ideally within visual and audible range so that you can listen and observe it when it does it. If you have the facility to record videos, onboard and ground based that would be even better too.

Also post close-up high-res, in-focus photos of your build once you've got the FC back in there.

Please also inspect the FC closely before re-installing it, and check for signs of debris and/or damage.

[hydra]

@danielolsman any update?

[DanO83]

@hydra no sorry, still lack of good weather and time

[robertgodlewski]

Hello everyone,

it seems that I have also experienced roll of death on H7EXTREME and bidirectional dshot. My quad fell into the water after sudden stop of all motors. I found it after 1,5h and surprisingly FC is working despite massive corrosion (but not working enough to fly again). This was i believe 6th fly on BF 4.1.6 and first with any problems at all. I have also noticed a lot of dshot errors during testing on the bench as well as problems with flashing blheli. Someone mentioned (in BF thread I think) that he had the same problems when camera was connected. I can confirm the same behaviour. I have also found that it was not limited to camera - any UART activity (eg. serial rx) causes problems with communication between FC and ESC. I was manage to reduce dshot errors to <0.5% on one only motors by soldering gnd signal wire (Tekko32 F3 35A do not have signal gnd pad) and twist it together with signal wire. HOWEVER. The are a lot of dshot errors during the flight (see DEBUG in attached log). BTFL_BLACKBOX_LOG_20200509_184405.zip

I hope it somehow helps to solve this problem. Now I am forced to look for new FC and i do not know what to do - I was really liking H7 for performance and features but for now it lacks in reliability.

[DanO83]

I recently did some test flights after converting my 6" H7extreme setup to a 7", first I wasn't willing to try bidir dshot. After a while I wanted to retry rpm filter so I enabled it. And it worked great! I thought i didn't changed any configuration, but i found out that I only was using the first gyro! It works for me, so i'm not going to do any further testing. Maybe this death roll issue has something to do with enabling both gyros

[hydra]

@robertgodlewski your issue sounds like DSHOT signal integrity. Do you have access to an oscilloscope so that you can view/capture/record the DSHOT signals sent by the FC and the DSHOT signals sent by the ESC?

You may find that some corrosion can be removed with appropriate solder flux, but do NOT try and ultrasonic clean the PCB as ultrasonic cleaning will kill the gyros, baro and mic.

[robertgodlewski]

@hydra I agree, SI could cause a dshot errors - no doubt about it. I have a scope and I can capture some signals in a free time - just give me more specifics what you want to see. But what about roll of death? You think it was also caused by SI? It seems unlikely to me.

Removing the corrosion is pointless in my opinion...

[hydra]

@robertgodlewski. I don't think roll of death is caused by signal integrity (SI). That feels more like ESC/Motor issue, especially in cases where the CPU still logs data. Roll of deaths can also occur if the CPU is reset due to brown-out condition on the 3v3 rail, in those cases it's clear as the logs are truncated and the startup tones can be heard again if a beeper is attached.

If you are able to capture FC -> ESC DSHOT control signals and resulting ESC -> FC DSHOT telemetry traces that would be great. Look for any inconsistency (noise, voltage, levels, slew, timing, etc) and attach scope traces here if you can.

[hydra]

I discovered a bug in BLHeli recently which affects the Tekko32 F3, details are here:

https://github.com/bitdump/BLHeli/issues/468

other people are also reporting this, which may be related.

https://github.com/bitdump/BLHeli/issues/464

From what I gather so far, the problem seems to be ESC related and not FC related as it happens on other FCs.

Affected users can should wait for updated BL Heli firmware and report bugs on the BLHELI issue tracker instead of here.

@joelucid you can close this issue now.