Bootloader: USB detection issues after update to timeout reset when USARTs defined in bootloader

[vierfuffzig]

board: matekf405-wing

it's one of the boards with multiple usarts defined in bootloader hwdef. after upgrading bl to https://github.com/ArduPilot/ardupilot/commit/3edf26dff9dab01c4010c9ba51d271a41cb8851c i do see random hangs when booting from USB (PC connect) only. bootloader gets stuck, LEDs freeze, no boot to runtime. booting from battery power shows no issues whatsoever and un-freezes the previously stuck bootloader immediately. removing USART defines from bootloader or reverting to a previous bootloader reliably solves above issues. my assumption is that latest bl upgrade to some extent interferes with connection detection handling if USARTs are defined in bl.

on a sidenote, the above mentioned commit added upgraded bootloader.bin files exclusively, but missed to rebuild .hex and .elf files too. already pinged @bugobliterator on discord respectively.

Version https://github.com/ArduPilot/ardupilot/commit/4b2552b32b9b124bd7544e9ffd02e4862f50aba4 and later

Hardware type MatekF405-Wing

[vierfuffzig]

https://github.com/ArduPilot/ardupilot/pull/17453 for reference

[bugobliterator]

@vierfuffzig just to confirm, if you externally power the board you reliably don't see hangup in bootloader, is that correct?

If that's the case I wonder if the chip is browning out. One thing I would try to confirm if that's the case is disable the LEDs in hwdef and try that firmware and see if that causes the issue.

[vierfuffzig]

@bugobliterator yes, boot from ext batt doesn‘t show issues as compared to USB. i can well try with LEDs disabled to make sure it‘s not a brownout. however LeDs have never been an issue on that board, reverting to previous bl or uncommenting UARTs solves hangs reliably. adding a 2A rated USB supply doesn‘t help. additionally on brownout i‘d rather expect a reboot-loop, what i do see is a hang-type behaviour though. will report back after tests with LEDs undefined.

[bugobliterator]

@vierfuffzig also what happens you connect both USB and ext batt at the same time?

[vierfuffzig]

@bugobliterator i‘m not quite sure how i‘ll manage to respect heisenberg‘s relation regarding „same time“, think i‘ll ask my wife to assist me ;)

[bugobliterator]

You can power up with brick, connect over usb ask for it to reboot.

[vierfuffzig]

@bugobliterator observations: 1.: adding ext power on reboot does not resolve issues 2.: current master's bootloader is NEVER identified correctly on fw-update through MP. 3.: removing LEDs solves identification issues, as well does removing USARTs. this does NOT seem to be power-related though. 4.: i'm seeing parameter resets after bl stuck situation.

bootloaders used for reference:

https://github.com/vierfuffzig/Chibios-miscellaneous/blob/master/MatekF405-Wing_bl_default_master.hex

https://github.com/vierfuffzig/Chibios-miscellaneous/blob/master/MatekF405-Wing_bl_noLEDs.hex

[bugobliterator]

@vierfuffzig thanks for your help with tracking down the issue. There are reports of the bootloader getting stuck everytime when loading heli firmware on this board. Can you check that? Also when you had LEDs disabled you had UART enabled, correct? If just disabling LEDs is fixing issue, it does point towards brownout. But this theory still just a suspect, I am not locked on to this just yet. If you can use multimeter to test VDD(3.3V supply) pin that'd be great. Also do you have access to debugger probes like STLink or JLink?

[vierfuffzig]

@bugobliterator correct, ONLY led disabled as compared to current master

can’t tell what difference fw vehicle type should make but will check on trad heli.

reverting latest changes does fix the issues even with adding periphereals (GPS + RC Rx) powered from USB so i personally am not convinced of the brownout theory, will continue testing anyways.

i think i have an ST link dongle, need to check;)

[andyp1per]

It would be ideal if you could pick up the revert then rebuild the bootloader for your board to see whether its sid's actual change or some prior change that is causing this

[bugobliterator]

Yes trad heli test would be useful.

The peripherals are powered via 5V line directly though, not the 3.3V line which is via the regulator. My suspicion is that particular regulator is not doing good enough job.

[bugobliterator]

It would be ideal if you could pick up the revert then rebuild the bootloader for your board to see whether its sid's actual change or some prior change that is causing this

@andyp1per its known that its not my particular change that's causing this, we haven't rebuilt bootloader for a long while, we have upgraded ChibiOS as well after last rebuild.

[vierfuffzig]

@bugobliterator @andyp1per i guess we do have separate issues that might partly interact. imho it‘s likely that current USB detection issues do relate to sid‘s commit. bl hangs due to periphereal pin interactions (UART mostly) have been an on-off-issue for years. all i can say is i hadn‘t seen any bl issues for quite some time until latest rebuild.

will report back after testing. btw no debug hardware. a few years ago i gave my st link to a friend who would make more use of it...

[vierfuffzig]

@bugobliterator @andyp1per interesting: reverting only https://github.com/ArduPilot/ardupilot/commit/3edf26dff9dab01c4010c9ba51d271a41cb8851c + rebuilding bootloader does NOT solve the issue!

can't spot any difference in behaviour with regard to vehicle type fw used so far (no difference in trad heli vs plane)

got my board back into a bl hang situation again, adding ext power on reboot definitely does not solve the issue. unplugging USB however does!

tested checking out https://github.com/ArduPilot/ardupilot/commit/ff2ae1d7d18c5d35d1f02e02bc5c4d7ee5e8fd49

• erasing existent bootloaders and rebuilding them prior to fw compilation

[bugobliterator]

@bugobliterator @andyp1per interesting: reverting only https://github.com/ArduPilot/ardupilot/commit/3edf26dff9dab01c4010c9ba51d271a41cb8851c + rebuilding bootloader does NOT solve the issue!

can't spot any difference in behaviour with regard to vehicle type fw used so far (no difference in trad heli vs plane)

got my board back into a bl hang situation again, adding ext power on reboot definitely does not solve the issue. unplugging USB however does!

tested checking out https://github.com/ArduPilot/ardupilot/commit/ff2ae1d7d18c5d35d1f02e02bc5c4d7ee5e8fd49

• erasing existent bootloaders and rebuilding them prior to fw compilation

Thanks for the report @vierfuffzig Do you have access to schematic of the board, what would be useful is if you can probe the 3.3V supply while stuck in bootloader and you transition from USB to Power Brick.

[vierfuffzig]

@bugobliterator can't spot voltage dips on 3v3 no matter if ext power added or USB only

what i found is that i can get the bl to crash by

• booting from ext power
• plugging in USB (PC connect) while in bootphase

[bugobliterator]

Thanks @vierfuffzig yeah this certainly gives a much better picture. Will get someone from dev team to check this using debugger.

[bugobliterator]

@vierfuffzig I have received this board, I will be trying to reproduce this issue soon.

[bugobliterator]

I am able to replicate the issue.

[vierfuffzig]

@bugobliterator that‘s great (sort of), thanks for reporting! any educated guesses what the root cause is…?

[bugobliterator]

Trying to attach debugger currently, if that fails, its back to basics GPIO toggle and printf debugging.

[bugobliterator]

well problem seems to go away when disabling LED and using them as SWD instead, that's not very helpful. Its going to be trickier to track this issue down that I had thought. I guess GPIO and printf debugging it is.

[tridge]

@vierfuffzig I think this PR likely fixes the problem: https://github.com/ArduPilot/ardupilot/pull/18122 confirmed .... I reproduced the issue, and rebuilding the bootloader with the chibios updates in #1822 fixes it

[github-actions[bot]]

Heads up @bugobliterator - the "ChibiOS" label was applied to this issue.

[vierfuffzig]

thanks @tridge i'll close this then. could you drop a few words on what the issue was just for completeness?