search

espressif / esptool

Espressif SoC serial bootloader utility
https://docs.espressif.com/projects/esptool
GNU General Public License v2.0
5.6k stars 1.39k forks source link

Can't reboot device using USB Serial JTAG when in DOWNLOAD_MODE (ESPTOOL-842) #970

Closed chipweinberger closed 1 month ago

chipweinberger commented 7 months ago

Update: summary of issue: if originally booted in DOWNLOAD_MODE, device stays stuck in DOWNLOAD_MODE. rebooting does not clear strapping pins. This only occurs with USJ. not UART.

Operating System

macOS 13.6.3

Esptool Version

esptool.py v4.6.2

Python Version

Python 3.9.6

Chip Description

esp32s3

Device Description

custom pcb

Hardware Configuration

I have a power-on reset circuit on my custom pcb.

Note: RST is connected to a debug header on the PCB. It's irrelevant to this issue / not used by my flash tool

Screenshot 2024-04-09 at 11 27 04 PM

How is Esptool Run

from python using subprocess

Full Esptool Command Line that Was Run

/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub --after hard_reset get_security_info

Esptool Output

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Reboot                                       

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub --after hard_reset get_security_info

esptool.py v4.6.2
Serial port /dev/cu.usbmodem101
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 48:27:e2:c9:0a:88
Enabling default SPI flash mode...
Flags: 0x00000000 (0b0)
Flash_Crypt_Cnt: 0x0
Key_Purposes: (0, 0, 0, 0, 0, 0, 12)
Chip_ID: 9
Api_Version: 0
Hard resetting via RTS pin...
None
None
 ___ _   _  ___ ___ ___ ___ ___  
/ __| | | |/ __/ __| __/ __/ __| 
\__ \ |_| | (_| (__| _|\__ \__ \ 
|___/\___/ \___\___|___|___/___/ 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Console                                      

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Waiting on /dev/cu.usbmodem101
found device!
Waiting for Console...

More Information

I have my own programming tool that then flashes then listens on the serial port. It does these steps:

  1. flashes firmware using write_flash
  2. reboots the chip using get_security_info --after hard_reset
  3. wait 1.5 seconds
  4. opens a python serial console using serial.Serial() & console.open()
  5. then waits to hear a magic phrase on the serial port

There are a few problems:

  1. the device never reboots. I know this because a buzzer plays a special tune on boot, and it never does
  2. step 5 fails. there is nothing heard on the serial port
  3. kind of odd, step 4 succeeds. at least, python says it does.

The only way to get it to reboot is to physically unplug the device and plug it in. then I hear the special tune and the serial console works again.

I Have Read the Troubleshooting Guide

chipweinberger commented 7 months ago

I've also tried rebooting using the run command. but no difference at all

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Reboot                                       

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub run

esptool.py v4.6.2
Serial port /dev/cu.usbmodem101
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 48:27:e2:c9:0a:88
Enabling default SPI flash mode...
Hard resetting via RTS pin...
None
None
 ___ _   _  ___ ___ ___ ___ ___  
/ __| | | |/ __/ __| __/ __/ __| 
\__ \ |_| | (_| (__| _|\__ \__ \ 
|___/\___/ \___\___|___|___/___/ 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Console                                      

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Waiting on /dev/cu.usbmodem101
found device!
Waiting for Console...
chipweinberger commented 7 months ago

when my device gets in this "stuck" state, i've tried invoking run or --after hard_reset get_security_info multiple times, to see if it would work a second, or third time.

everytime I call the command it says it "succeeds". 

chipweinberger@Chips-MacBook-Pro jflash % /Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub run
esptool.py v4.6.2
Serial port /dev/cu.usbmodem101
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 48:27:e2:c9:0a:88
Enabling default SPI flash mode...
Hard resetting via RTS pin...

but my device does not actually reboot. it appears to do nothing.

only physically pulling the power cord will reboot it.

dobairoland commented 7 months ago

Hi @chipweinberger. Of course it reports success. There is no confirmation or check that the reset have been successful. The appropriate pin has been toggled using RTS of the serial interface. But there is no actual confirmation that the chip has been reseted.

I'm afraid you made a mistake in the PCB design and that is why the chip cannot be reseted automatically.

chipweinberger commented 7 months ago

Thanks for the response, but I don't understand.

In theory could have an ESP32 with only 4 pins connected:

Is this sufficient for USB Serial JTAG to reset the chip?

What other PCB traces / design would you need?

chipweinberger commented 7 months ago

I just tried using soft_reset, but no difference, device does not enter my application.

chipweinberger@Chips-MacBook-Pro jflash % /Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub --after soft_reset get_security_info

esptool.py v4.6.2
Serial port /dev/cu.usbmodem101
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 48:27:e2:c9:0a:88
Enabling default SPI flash mode...
Flags: 0x00000000 (0b0)
Flash_Crypt_Cnt: 0x0
Key_Purposes: (0, 0, 0, 0, 0, 0, 12)
Chip_ID: 9
Api_Version: 0
Soft resetting...
chipweinberger commented 7 months ago

@igrr curious your thoughts here as well.

chipweinberger commented 7 months ago

I've found the underlying cause.

when my device is in this "stuck" state, refusing to reboot, running idf.py monitor gives this output

chipweinberger@Chips-MacBook-Pro fcc-testing % idf.py monitor           
/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/tools/check_python_dependencies.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Executing action: monitor
Serial port /dev/cu.usbmodem101
Connecting...
Detecting chip type... ESP32-S3
Running idf_monitor in directory /Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/fcc-testing
Executing "/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/tools/idf_monitor.py -p /dev/cu.usbmodem101 -b 115200 --toolchain-prefix xtensa-esp32s3-elf- --target esp32s3 /Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/fcc-testing/build/tusb_midi.elf -m '/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python' '/Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/tools/idf.py'"...
--- idf_monitor on /dev/cu.usbmodem101 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x3 (DOWNLOAD(USB/UART0))
Saved PC:0x40048ee5
0x40048ee5: recv_packet in ROM

waiting for download

This is the cause!

Straight from the espressif factory, I always have to put my esp32s3 into DOWNLOAD MODE using GPIO 0. This is the only way I know to get an esp32s3 straight from the factory to listen over usb serial jtag.

If I do not first put my device into DOWNLOAD MODE, then my reboot command works.

This is tricky though, because I need DOWNLOAD MODE to program my device. Yet, it prevents rebooting into the application without physically pulling the power cord.

I really don't want to have to physically the power cord during my manufacturing process. Is there any way to program an esp32s3 then reboot into my application using USB Serial JTAG, programmatically?

radimkarnis commented 7 months ago

Hi @chipweinberger,

In theory could have an ESP32 with only 4 pins connected: usb+ usb- 3.3v gnd

Yes, the USB-Serial/JTAG peripheral has a state machine that detects the DTR and RTS transitions and resets the chip. These four connections should be enough for it to work.

I always first put my device into DOWNLOAD MODE using GPIO 0.

That means you manually put the device into download mode by physically pulling the GPIO0 strapping pin down. During bootup, this gets stored in the strapping pin register. If that isn't cleared, USB-Serial/JTAG might not be able to leave the download mode.,

You can try updating esptool to the latest master, we have recently added clearing of the boot control register during reset: https://github.com/espressif/esptool/commit/1c355f9fad13f214fde0566fa307a333a4413697

chipweinberger commented 7 months ago

@radimkarnis , that looks like exactly the issue I am having. yay.

I'm very close to production at this point, so I don't want to switch to master. I guess I could cherry-pick this commit. let me try that.

chipweinberger commented 7 months ago

@radimkarnis , I'm currently on v5.1, and can't seem to find that file esptool/targets/esp32s3.py.

Edit: found it here /Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/lib/python3.9/site-packages/esptool/targets/esp32s3.py

Would you be able to give me a little guidance?

/ how long would I need to wait for this to hit v5.1?

chipweinberger commented 7 months ago

https://github.com/espressif/arduino-esp32/issues/6762

dobairoland commented 7 months ago

Ok, sorry. I've missed that you are using USB Serial JTAG.

dobairoland commented 7 months ago

Esptool installed in the ESP-IDF environment. After activating the environment, you can upgrade it by running pip install with the path to the cloned esptool master branch.

how long would I need to wait for this to hit v5.1?

I don't have an exact estimate. esptool 4.8 will be probably released in 3-6 Months.

chipweinberger commented 7 months ago

I've tried patching in the change, but it still does not reboot my device!

    def hard_reset(self):
        uses_usb_otg = self.uses_usb_otg()
        if uses_usb_otg:
            self._check_if_can_reset()

        try:
            print("Clearing RTC_CNTL_FORCE_DOWNLOAD_BOOT_MASK")
            # Clear force download boot mode to avoid the chip being stuck in download mode after reset
            # workaround for issue: https://github.com/espressif/arduino-esp32/issues/6762
            self.write_reg(
                self.RTC_CNTL_OPTION1_REG, 0, self.RTC_CNTL_FORCE_DOWNLOAD_BOOT_MASK
            )
        except Exception:
            # Skip if response was not valid and proceed to reset; e.g. when monitoring while resetting
            pass

        print("Hard resetting via RTS pin...")
        HardReset(self._port, uses_usb_otg)()

Logs confirm my patch is being called.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Reboot                                       

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/Volumes/User/MBP-Google-Drive/jamcorder/firmware/.espressif/python_env/idf5.1_py3.9_env/bin/python /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/cu.usbmodem101 --chip esp32s3 --no-stub --after hard_reset get_security_info

esptool.py v4.6.2
Serial port /dev/cu.usbmodem101
Connecting...
Chip is ESP32-S3 (revision v0.2)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 48:27:e2:c9:0a:88
Enabling default SPI flash mode...
Flags: 0x00000000 (0b0)
Flash_Crypt_Cnt: 0x0
Key_Purposes: (0, 0, 0, 0, 0, 0, 12)
Chip_ID: 9
Api_Version: 0
Clearing RTC_CNTL_FORCE_DOWNLOAD_BOOT_MASK
Hard resetting via RTS pin...
None
None
 ___ _   _  ___ ___ ___ ___ ___  
/ __| | | |/ __/ __| __/ __/ __| 
\__ \ |_| | (_| (__| _|\__ \__ \ 
|___/\___/ \___\___|___|___/___/ 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

                               Console                                      

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Waiting on /dev/cu.usbmodem101
found device!
Waiting for Console...

idf.py monitor is still in download mode

--- idf_monitor on /dev/cu.usbmodem101 460800 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x3 (DOWNLOAD(USB/UART0))
Saved PC:0x40048d06
0x40048d06: uart_rx_one_char in ROM

waiting for download
radimkarnis commented 7 months ago

I have tried to reproduce this and can confirm this happens.

If download mode is entered by pulling the BOOT strapping pin low (GPIO0 on ESP32-S3), USB-Serial/JTAG cannot exit the download mode.

I am not sure if this is expected behavior. We will look into this and see if this can be patched.

chipweinberger commented 7 months ago

please let me know as soon as you know! thanks

until then i've begrudgingly added a physical replug step to my manufacturing line.

chipweinberger commented 7 months ago

a solution

it would be ideal if espressif chips listened to serial jtag from the factory without needing download mode

that would save me another manufacture step.

what firmware do they ship with? why don't they listen?

dobairoland commented 7 months ago

If download mode is entered by pulling the BOOT strapping pin low (GPIO0 on ESP32-S3), USB-Serial/JTAG cannot exit the download mode.

And I think we cannot do anything about this. The linked patch turns off the possibility to force of entering download mode by means of a register. If we turn this off then the content of the strapping register (including the captured GPIO0) applies. The strapping register is read only so we cannot do anything about the fact that the chip was put into download mode through GPIO0.

We would need a similar "Force normal boot mode" register but there isn't any. I'm sorry, I could not find anything, therefore, I'm afraid that this is not possible.

what firmware do they ship with? why don't they listen?

I think all flash chips are erased and there is no firmware. To the best of my knowledge, the reason is that erasing the flash before flashing takes considerable time and this would delay the production line.

To me, flashing through UART in production even thought the application will be using Serial JTAG or OTG sounds the most reasonable solution.

chipweinberger commented 7 months ago

thanks for looking. that's too bad 🙁

radimkarnis commented 7 months ago

During power-on-reset, RTC watchdog reset, brownout reset, analog super watchdog reset, and crystal clock glitch detection reset (Chapter 7 Reset and Clock in the S3 TRM), hardware captures samples and stores the voltage level of strapping pins as strapping bit of “0” or “1” in latches, and holds these bits until the chip is powered down or shut down. Software can read the latch status (strapping value) from the register GPIO_STRAPPING.

The issue here might be that the USB-Serial/JTAG peripheral resets the chip (when an RTS transition is detected), but the reset doesn't qualify for a new reading of the strapping pin voltages (so the GPIO0 reading is being held and the chip enters the download mode again).

We will think about patching this in esptool to invoke another kind of reset (possibly RTC watchdog reset) - that could qualify for a new voltage reading on the strapping pins.

However, this will take some consideration and time. I am leaving this open until then.

chipweinberger commented 7 months ago

thanks for your updated thoughts

could I temporarily fix this in esptool myself in the interim?

is there some code i could change?

Jason2866 commented 6 months ago

Maybe this works. esptool.py --no-stub --chip esp32s3 --port <used port> flash_id

chipweinberger commented 5 months ago

@radimkarnis , any progress?

I'd love a patch, because I'll be doing production in 1 month.

radimkarnis commented 5 months ago

I don't think we can issue a proper "patch" due to above-mentioned reasons, more like a "hacky" reset approach.

Nevertheless, there is no progress on this task. We will take another look and see if we can up the priority.

chipweinberger commented 5 months ago

right - I meant towards using RTC watchdog reset, or similar.

Thanks for thinking about it again.

Also curious - is something that would get fixed in a new silicon revision?

radimkarnis commented 1 month ago

Hello @chipweinberger, a possible fix has been merged. This is a new experimental reset approach. Please try to verify if this commit solves the issue. The reset should kick in automatically in USB-Serial/JTAG mode.

chipweinberger commented 1 month ago

Thanks.