Seeed Studio ESP32C3 based SOTACAT causes KX to lock up on boot: UART sending boot messages to KX??

[brianmathews]

SOTACAT modules based on the Seeed Studio ESP32C3 MCU causes KX2/KX3 radios to lock up if the SOTACAT is connected to the radio when booting. It appears that the ESP32C3 is emitting a bunch of boot messages into the KX via the UART and that these random characters get the KX into a locked state that can only be fixed by a KX power cycle.

• If the ESP32C3 is booted first, and then attached to the KX radio there is no problem.
• The problem does not seem to happen on Wemos LOLIN32 Lite based ESP32 boards
• I tried using the PlatformIO / Espressif "Run MenuConfig" tool and tried changing "Bootloader Config -> Bootloader Log verbosity -> None" and other variations and this did not solve the issue.
• There are two active UARTs in SOTACAT: the one connected to the USB connector which are sent to the host computer when debugging, and the UART connected to the KX radio. I expect boot messages to go to the USB UART and not the KX UART, especially since the KX UART is not configured in the Firmware to send anything until after booting. So I'm confused what is happening.

If turning off bootloader messages doesn't solve the issue, then I am confused why the KX is locked up. We may want to attach a serial port sniffer between the radio and the KX.

[brianmathews]

I have tried the following, without success, to try and get rid of boot messages being sent to the KX via UART:

1. The normal way is to pull GPIO15 low at boot to prevent boot messages. We don't have a GPIO15 available on the Seeed Studio XAIO ESP32-C3.
2. I tried menuconfig to turn off boot logging. Has no effect. PlatformIO has its own build approach.
3. I tried defining build_flags in my platformio.ini file but it causes compiler warnings about redefining settings in sdkconfig.h: build_flags = -DCONFIG_LOG_BOOTLOADER_LEVEL_NONE
4. I tried using a sdkconfig.defaults file with the setting “CONFIG_BOOTLOADER_LOG_LEVEL_NONE=y” and "=1" and that seems to work, but the KX still locks up. I don't have an in-line UART monitor between the ESP32 and the KX radio so it is hard for me to confirm why this did not work.

I give up for now.

[jeffkowalski]

I can give it a shot if someone can give me reliable steps to reproduce. Truthfully, I've never seen the error here. I can plug my cat into the radio at any time, with or without a usb connection to computer. What I'd need to know is - starting from everything off and disconnected, which sequence of steps causes the error?

[brianmathews]

Hmmm. Interesting. I think this might be a KX3 only issue. I used to develop on my KX2 with internal battery, but kept forgetting to turn off the KX2 and running my batteries into the ground. So I recently switched to the KX3 which has a power supply on the bench. Come to think of it, I only started noticing the lock-ups on the KX3... Hmmm. The KX2 firmware might be better at rejecting random serial data.

[jeffkowalski]

I see that IDF uses the default UART for logging messages, but perhaps this is reconfigurable.

[brianmathews]

I think it is reconfigurable, but not in our software since the bootloader that is causing the issues is loaded even before our code is started. To reconfigure it the ESP documentation talks about the methods above, and in particular the menuconfig program which is not working for me. The other sledgehammer approach is to take the open source bootloader code from Espressif and build a custom version of the bootloader ourselves. Yuck...

[poynting]

Moving the TX data pin one pin over to GPIO7 also fixes the problem. I've confirmed this on my breadboard SOTAcat, and it makes development quicker for me (and saves a lot of button presses rebooting the KX3). With the default pins, after adding buffers which remove the need for inversion I was starting to get ERR-FW2 messages on the KX3 regularly which I didn't like. With the buffers and pin changes, I don't see any hangups or error messages, and I also see fewer retries scrolling by in the debug log.

I have some other IO changes I'd like to make as well for the next PCB revision - this doesn't feel like the best place to discuss next revision hardware and the implications of changes to the firmware -- is there another spot we can discuss? Should I Open a new issue for that conversation?

[brianmathews]

Yes, I think most versions of the ESP32 allow the UART to be remapped to a different pin. I have to look at the docs for the C2. I vaguely remember there were certain ESP32 models that had restrictions on remapping of the UART. If the C3 allows it that would be a very simple solution to our problem and I am embarrassed I didn't think of that - that is a very clean and simple solution.

[poynting]

To clarify - I remapped the application UART pin to GPIO7 in software and physically moved the wire over one pin. This is a simple solution but isn't compatible with current hardware. It looks like we should be able to suppress 2nd stage boot loader and application messages via software, but the ROM boot loader requires an eFuse to be set in order to disable messages. See the datasheet section 2.6.2 (images below).

To suppress the UART output on the second stage bootloader, it looks like we use "Bootloader Config -> Bootloader Log Verbosity"

And to prevent UART output from the application, we use Component config → ESP System Settings → Channel for console output.

To deal with the first stage (ROM) bootloader, we have to use eFuses and/or strapping pins. The options are to disable output entirely and permanently, or to disable output based on a pin state at power-up. In the SEED XIAO, GPIO9 is pulled up by a resistor in the module (and can alternatively be held low via the boot button). GPIO8 is floating currently, and we use it as the cathode of the amber LED.

So, if we want to disable the boot messages from the ROM boot loader, it appears we could lightly pull up or down GPOP9, while setting the EFUSE_UART_PRINT_CONTROL to 1 or 2, depending on whether we want to pull it up or down. I would expect it would be best to pull it down externally for minimal power draw in sleep mode. This selection can be changed in menuconfig under "Boot ROM Behavior."

[jeffkowalski]

Justin,

You're a super-star for finding a solution so quickly. Just so I understand, when you say "current hardware" in your first paragraph, do you simply mean the as-built SOTAcat? If we were to simply change wiring per your recommendation, would all be resolved? Or do we need to go deeper into the board of the seeed xiao itself to find the relevant pin? Right now, I think there are very few of us that have "current hardware", and so I would recommend that we take "current" very lightly and be adaptable to any change you recommend.

Jeff

On Wed, Mar 6, 2024 at 3:47 PM Justin McAllister @.***> wrote:

To clarify - I remapped the application UART pin to GPIO7 in software and physically moved the wire over one pin. This is a simple solution but isn't compatible with current hardware. It looks like we should be able to suppress 2nd stage boot loader and application messages via software, but the ROM boot loader requires an eFuse to be set in order to disable messages. See the datasheet section 2.6.2 (images below).

To suppress the UART output on the second stage bootloader, it looks like we use "Bootloader Config -> Bootloader Log Verbosity" image.png (view on web) https://github.com/SOTAmat/SOTAcat/assets/6610131/4fed61ff-c4b1-4a03-8f6b-b8bba320776f

And to prevent UART output from the application, we use Component config → ESP System Settings → Channel for console output. image.png (view on web) https://github.com/SOTAmat/SOTAcat/assets/6610131/b9d6d709-606f-4446-b6c8-735f2f9e70f4

To deal with the first stage (ROM) bootloader, we have to use eFuses and/or strapping pins. The options are to disable output entirely and permanently, or to disable output based on a pin state at power-up. In the SEED XIAO, GPIO9 is pulled up by a resistor in the module (and can alternatively be held low via the boot button). GPIO8 is floating currently, and we use it as the cathode of the amber LED.

So, if we want to disable the boot messages from the ROM boot loader, it appears we could lightly pull up or down GPOP9, while setting the EFUSE_UART_PRINT_CONTROL to 1 or 2, depending on whether we want to pull it up or down. I would expect it would be best to pull it down externally for minimal power draw in sleep mode. This selection can be changed in menuconfig under "Boot ROM Behavior."

image.png (view on web) https://github.com/SOTAmat/SOTAcat/assets/6610131/ca216e0e-1d96-4d15-a213-ba731d9413ad image.png (view on web) https://github.com/SOTAmat/SOTAcat/assets/6610131/bd7d002a-25f4-4c7a-9854-bfbafe250b86

— Reply to this email directly, view it on GitHub https://github.com/SOTAmat/SOTAcat/issues/2#issuecomment-1982057805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN4MD4YQIWJLTGX2U6EODYW6TKXAVCNFSM6AAAAABDTTSPISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBSGA2TOOBQGU . You are receiving this because you commented.Message ID: @.***>

[poynting]

I believe the most conservative hardware design approach moving forward is to avoid using the hardware UART pins tied to the ROM bootloader. Since we're not currently pin-constrained, simply moving the UART TX line to a different pin for future hardware is a very easy approach.

However, I also understand that there is a handful of current and in-progress hardware with the initial pinout per Brian's current documentation. My PCB (which I call v0.2 as v0.1 never went to fab) shares the same pinouts with Brian's initial schematic. So, to continue to support that hardware, I have also developed a workaround. I've been able to completely disable extra bootloader traffic using the two menuconfig approaches above, as well as one eFuse change.

Changing this option allows disabling the ROM bootloader UART output by tying GPIO8 (module pin 9, our current Amber/Red LED Cathode) to ground, or enabling the ROM bootloader UART output by tying GPIO8 to 3.3V during bootup.

I've verified with a scope that this group of options does disable all output from XIAO module pin 7 / GPIO21.

[brianmathews]

I think this is fixed with your suggestions. Amazing. I tried so many of those settings and you nailed it! Nice work @poynting ! I was not able to lock up my KX3 anymore after several tries. I made the 3 changes you suggested in Menuconfig, which get stored in the sdkconfig.seeed_xiao_esp32c3 file (and I made an exception in the .gitignore to allow it back into the project).

[jeffkowalski]

Fix reportedly cures KX3 lockup and I can confirm no ill-effect on my KX2. Closing.