Cannot flash without going into bootloader mode

[xtrinch]

While it's perfectly doable to put the board in bootloader mode and then flash, it would be really great to be able to flash it without that, like any other esp32 boards. The board isn't in sleep mode or anything, and I can even talk to it via webusb, but I can't flash it.

I'm getting this error:

CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: /dev/ttyACM0
Uploading .pio/build/esp32s2doit-devkit-v1/firmware.bin
esptool.py v3.1
Serial port /dev/ttyACM0
Connecting...
Traceback (most recent call last):
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 4582, in <module>
    _main()
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 4575, in _main
    main()
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 4074, in main
    esp = esp or get_default_connected_device(ser_list, port=args.port, connect_attempts=args.connect_attempts,
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 121, in get_default_connected_device
    _esp.connect(before, connect_attempts)
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 629, in connect
    last_error = self._connect_attempt(mode=mode, esp32r0_delay=False, usb_jtag_serial=usb_jtag_serial)
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 590, in _connect_attempt
    self.bootloader_reset(esp32r0_delay, usb_jtag_serial)
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 553, in bootloader_reset
    self._setDTR(False)  # IO0=HIGH
  File "/home/xtrinch/.platformio/packages/tool-esptoolpy/esptool.py", line 499, in _setDTR
    self._port.setDTR(state)
  File "/home/xtrinch/.platformio/penv/lib/python3.8/site-packages/serial/serialutil.py", line 603, in setDTR
    self.dtr = value
  File "/home/xtrinch/.platformio/penv/lib/python3.8/site-packages/serial/serialutil.py", line 473, in dtr
    self._update_dtr_state()
  File "/home/xtrinch/.platformio/penv/lib/python3.8/site-packages/serial/serialposix.py", line 715, in _update_dtr_state
    fcntl.ioctl(self.fd, TIOCMBIC, TIOCM_DTR_str)
OSError: [Errno 71] Protocol error
*** [upload] Error 1

Any ideas? Any help would be greatly appreciated, even if only to point me in the direction of what this error is and where do I start looking.

[chegewara]

Yes. The problem is that espressif changed the way it is possible to flash over native USB and i need to update library.

[chegewara]

Should work now with master and v1.3.3

[xtrinch]

Afraid not, still the same error :(

[chegewara]

Strange, it works to me now.

All you have to do is to implement CDC code, so using CDC example is best way to test.

[oliverparis]

I can confirm I'm seeing basically the same issue with this using the CDC example and v1.3.3.

...
Uploading .pio\build\esp32s2doit-devkit-v1\firmware.bin
esptool.py v3.1
Serial port COM20
Connecting....
Traceback (most recent call last):
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 4582, in <module>
    _main()
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 4575, in _main
    main()
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 4076, in main
    before=args.before)
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 121, in get_default_connected_device
    _esp.connect(before, connect_attempts)
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 629, in connect
    last_error = self._connect_attempt(mode=mode, esp32r0_delay=False, usb_jtag_serial=usb_jtag_serial)
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 596, in _connect_attempt
    self.sync()
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 487, in sync
    timeout=SYNC_TIMEOUT)
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 425, in command
    self.write(pkt)
  File "C:\Users\Oliver\.platformio\packages\tool-esptoolpy\esptool.py", line 388, in write
    self._port.write(buf)
  File "c:\users\oliver\.platformio\penv\lib\site-packages\serial\serialwin32.py", line 323, in write
    raise writeTimeoutError
serial.serialutil.SerialTimeoutException: Write timeout
*** [upload] Error 1
====================================================================== [FAILED] Took 15.66 seconds ======================================================================The terminal process "C:\Users\Oliver\.platformio\penv\Scripts\platformio.exe 'run', '--target', 'upload'" terminated with exit code: 1.

I was going to try the same test using Arduino IDE, and noticed that the library.properties file still shows V1.3.2 so it wont update easily through library manager. Not tried manually updating the library yet.

Does it matter that both Xtrinch and I are using PIO? Is the CDC upload only supported in Arduino IDE at the moment?

[chegewara]

Question, did you change Upload Mode to internal?

[oliverparis]

OK, just manually updated the library in Arduino IDE and it works as expected - can program without manually entering bootloader mode. Upload Mode Internal must be set.

So seems that the only issue is actually that this isn't working with PIO?

[chegewara]

With this PR PIO should work fine soon and no longer needed to change local arduino files: https://github.com/espressif/arduino-esp32/pull/5475

With PIO is problem with arduino-esp32 compatibility (probably).

[oliverparis]

Great! That PR will make the PIO environment setup much less fiddly. Thank you.

Been looking more into the CDC upload again and I agree, this is not a problem with this library, it is related to the PIO Arduino ESP32 implementation. I get the same results using the standard ESP32 Arduino USB example. I will raise an issue over on that project, explain my findings and see what they say.

As a workaround for now @xtrinch, I found that if you remove any specific upload port from your platformio.ini, attempt upload, it will fail but the ESP will go into bootloader mode, then attempt the upload a second time, it will successfully upload and reboot back into your program.

[xtrinch]

@oliverparis thanks, but I think our errors differ, my board always throws protocol error, not timeout, and it never works. Also, my board is fully awake and I can read its output via the serial monitor.

[chegewara]

I know it may be strange, but for some reason it also depends what code is flashed. For example i had issue and many errors trying to flash over USB when old CDC example was flashed. That example has been using USB serial and Serial1. I dont know exactly what caused the problem, but when i removed Serial1 from example it suddenly started working. No more issue with flashing over USB.

[xtrinch]

I don't use regular Serial in my code.

Also I logged the DTS & RTS signals in the tud_cdc_line_state_cb callback, and they are nowhere near correct, it mostly just receives DTS=1,RTS=1 continously. Almost like something is interfering with the setting of these signals from esptool.

But there isn't any problem setting them when connecting via a terminal, then it just receives DTS=1,RTS=1 once.

[chegewara]

Currently arduino-esp32 and esptool is using baudrate 1200 to signal it will going into bootload mode. It is the way espressif decided to workaround it for now. No longer DTR/RTS are used for that.

[xtrinch]

Odd, that makes me a bit confused. My flashing routine crashes at the DTR sending, in the following line: https://github.com/espressif/esptool/blob/master/esptool.py#L572 I don't see any baud rate changing in that piece of code?

[chegewara]

Maybe you are still using old esptool. https://github.com/chegewara/EspTinyUSB/blob/master/src/classes/cdc/cdcusb.cpp#L215-L217

I think it is in arduino-esp32 master branch and you have to update tools by running get.py or get.exe.

Here is code from arduino-esp32 CDC responsible to enter bootload mode: https://github.com/espressif/arduino-esp32/blob/4ada3f5804e8165680141653377826d30a4ca364/cores/esp32/USBCDC.cpp#L231-L232

[xtrinch]

Oh, I didn't realize they have their own version of esptool.

[env:esp32s2doit-devkit-v1]
platform = espressif32
platform_packages = 
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32#master
  platformio/tool-esptoolpy @ ~1.30100

So now I'm running the master branch of arduino-esp32 and if I uncomment this line https://github.com/chegewara/EspTinyUSB/blob/991acff150f33b3d81dfac4999d24590d6b984ff/src/classes/cdc/cdcusb.cpp#L205 I can flash it (although every second time, like @oliverparis suggested)

[chegewara]

Yes. The first one is switching into bootload mode and second just perform update. Its because old esptool is not using 1200 baudrate, instead DTR/RTS.