With v2.2 baud rate maxes out at 230'400

[marcelstoer]

I used to be able to flash a WeMos D1 mini (CH340G on macOS) with 921'600.

esptool.py v2.0.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0240
Compressed 472084 bytes to 305673...
Wrote 472084 bytes (305673 compressed) at 0x00000000 in 4.5 seconds (effective 838.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

With esptool.py v2.2 the very same write_flash operation fails (same options, device, driver and binary):

esptool.py v2.2
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...

A fatal error occurred: Timed out waiting for packet header

The max baud that works is 230'400.

[projectgus]

Thanks for reporting this, @marcelstoer .

Are you able to please verify something for me? If you check out commit 81e6108f2b2dd840bc649ae92f875d7deebdfa25 and test, and then check out commit b813b373574a6a00092af5a47f4e7d62d4d7d7ae and test, does the former succeed and the latter fail? This will confirm what I suspect caused the regression.

That commit is supposed to make baud rate (crystal) inaccuracy in the hardware less of a problem, but it's possible that it's not working as expected (or not for all hardware)...

[marcelstoer]

If you check out commit 81e6108 and test,

That one seems already "broken":

esptool.py v2.2-dev
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...

A fatal error occurred: Invalid head of packet ('\xf4')

Turns out even v2.1 suffers from this. The behavior is different though:

esptool.py v2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0240
Compressed 472084 bytes to 305673...
^CTraceback (most recent call last):
  File "esptool.py", line 2524, in <module>
    _main()
  File "esptool.py", line 2517, in _main
    main()
  File "esptool.py", line 2276, in main
    operation_func(esp, args)
  File "esptool.py", line 1791, in write_flash
    blocks = esp.flash_defl_begin(uncsize, len(image), address)
  File "esptool.py", line 63, in inner
    return func(*args, **kwargs)
  File "esptool.py", line 498, in flash_defl_begin
    struct.pack('<IIII', write_size, num_blocks, self.FLASH_WRITE_SIZE, offset))
  File "esptool.py", line 273, in check_command
    val, data = self.command(op, data, chk)
  File "esptool.py", line 254, in command
    p = self.read()
  File "esptool.py", line 220, in read
    return next(self._slip_reader)
  File "esptool.py", line 1560, in slip_reader
    read_bytes = port.read(1 if waiting == 0 else waiting)
  File "/usr/local/lib/python2.7/site-packages/serial/serialposix.py", line 472, in read
    ready, _, _ = select.select([self.fd, self.pipe_abort_read_r], [], [], timeout.time_left())
KeyboardInterrupt

It was stuck after "Compressed 472084 bytes to 305673..." until I killed it. Looks like 2.0.1 is the last release that works in my setup.

I have to confess that I don't fully trust my test bed anymore, though. git checkout v2.0.1 rewinds back to ffde3f0 and all is well. git checkout ffde3f0 is supposed to do the same but flashing is stuck after the compression step.

[projectgus]

OK, thanks for the update. Recently I've mostly been running the ESP8266 automated tests with CP2102-based boards. CH340Gs have never been very stable for me at higher baud rates. But if they're more stable on the older releases, we should fix this.

I have to confess that I don't fully trust my test bed anymore, though. git checkout v2.0.1 rewinds back to ffde3f0 and all is well. git checkout ffde3f0 is supposed to do the same but flashing is stuck after the compression step.

That's odd, they should definitely be the same code. You can confirm with md5sum esptool.py or something like that. Is it possible the failures are intermittent in a semi-predictable way (maybe the CH340 needs a hard power cycle before it comes back)? Or something heat related (some of the CH340s seem to have fairly substantial clock/baudrate drift compared to the ESP8266)?

If you're feeling adventurous, you could try git bisect v2.2 v2.0.1 esptool.py and bisect to look for the commit which reliably starts failing at higher rates.

Otherwise I'll have access to my collection of ESP8266 boards next week and I'll try to reproduce this with a CH340G.

[kaybee335]

I have exactly the same problem as described with 8266 FeatherWing but with an interesting twist.

Under Win10 with esptool running in a python3 virtual environment I can program the 8266 at the baud rate of 460800.

Then again Win10 but under the Ubuntu Windows Subsystem for Linux and an identical python3 virtual environment it fails at 460800 when configuring the flash size but succeeds at 230400.

Same laptop, same USB cable, same 8266 for both configurations.

So apparently it's not just the ESP board but also how you get to it that affects the issue. There's something subtle on the timing there :-)

BTW - Thank you for the Xmas present of making esptool.py python3 compatible. Not sure when it happened but I just got rid of the last legacy python environment on my development PC!!!

[ygator]

See if #250 solves it.

[kaybee335]

@ygator - not sure if that suggestion was for me but just reducing the baud rate to 230400 works for me and I can just set that using the esptool command line flags.

[ygator]

@kaybee335 I was just wondering if you did #250 if you could get 460800 baud to work.

[kaybee335]

@kaybee335 I was just wondering if you did #250 if you could get 460800 baud to work.

I'll give it a try when I get a chance. I now have a working build configuration so I can get back to the micropython stuff I am actually supposed to be doing. I don't use esptool all that often (only for updating micropython from pre-built binaries). I use adafruit-ampy to upload my actual micropython code and it seems to run just fine at the default baud rate so my primary use case is using adafruit-ampy with just occasional use of esptool so I can live with the slower baud rate solution.

I will get to it in the next couple of days and post results once I figure out how to do it without borking my existing working configuration (looks like the change is pretty straightforward so I'm not too worried).

I'll also check what happens with an ESP32 Feather Huzzah32 to see if it exhibits the same behaviour as the ESP8266 Feather Huzzah (with and without mod).

[projectgus]

Hi @marcelstoer,

Have fixed an issue that may be the same as this one. Can you please try the latest master branch (with commit linked above) and let me know if it fixes the problem.

Angus

[ifitshow]

Hi @projectgus

23:00:54 Build of configuration Default for project FitShow make flash Flashing binaries to serial port /dev/tty.SLABUSBtoUART (app at offset 0x10000)... esptool.py v2.2 Connecting........ Chip is ESP32D0WDQ6 (revision 1) Uploading stub... Running stub... Stub running... Changing baud rate to 921600 Changed. Configuring flash size... Compressed 21424 bytes to 12580...

Writing at 0x00001000... (100 %) A fatal error occurred: Failed to write compressed data to flash after seq 0 (result was C100) make: *** [flash] Error 2

23:01:00 Build Finished (took 5s.710ms)

=====================================================================

23:02:17 Build of configuration Default for project FitShow make flash Flashing binaries to serial port /dev/tty.SLAB_USBtoUART (app at offset 0x10000)... esptool.py v2.1 Connecting........__ Chip is ESP32D0WDQ6 (revision 1) Uploading stub... Running stub... Stub running... Changing baud rate to 921600 Changed. Configuring flash size... Compressed 21424 bytes to 12580...

Writing at 0x00001000... (100 %) Wrote 21424 bytes (12580 compressed) at 0x00001000 in 0.2 seconds (effective 1128.5 kbit/s)... Hash of data verified. Compressed 1230544 bytes to 715055...

Writing at 0x000bc000... (100 %) Wrote 1230544 bytes (715055 compressed) at 0x00010000 in 11.2 seconds (effective 879.5 kbit/s)... Hash of data verified. Compressed 3072 bytes to 97...

Writing at 0x00008000... (100 %) Wrote 3072 bytes (97 compressed) at 0x00008000 in 0.0 seconds (effective 2980.0 kbit/s)... Hash of data verified.

Leaving... Hard resetting...

23:02:36 Build Finished (took 19s.234ms)

[projectgus]

Hi @ifitshow ,

Can you please try the master branch with the possible fix for this issue and let us know if it works then (see the comment directly above your comment).

Angus

[baycom]

Same problem under MacOS X: `Flashing binaries to serial port /dev/tty.SLAB_USBtoUART (app at offset 0x30000)... esptool.py v2.2 Connecting........___ Chip is ESP32D0WDQ6 (revision 1) Uploading stub... Running stub... Stub running... Changing baud rate to 921600 Changed. Configuring flash size... Auto-detected Flash size: 4MB Flash params set to 0x0221 Compressed 17536 bytes to 10504... Writing at 0x00001000... (100 %) A fatal error occurred: Failed to write compressed data to flash after seq 0 (result was C100)

make: *** [flash] Error 2`

It does not make any difference if data is compressed or not.

Reverting back to the old version by git checkout 754f81aa35a76421f9726b74af7ca2d9313b6230 esptool.py fixes the problem.

[projectgus]

Hi @baycom,

If you try the just relased v2.2.1, does this work (git fetch origin; git checkout v2.2.1)? For more details, please see the comment above yours.

Angus

[paulreimer]

I just tried the latest esp-idf master (2a10ffce5fd17761c420012bf1a53c84194febfb) and I have the same problem with my board.

I am also using macOS, and I was also programming my board at 921600. It has a CP2102 but not the faster CP2102n. I also have a dev board using FT2232H at 2MBaud and the latest esptool update in esp-idf broke that also.

I can confirm that the esptool v2.2.1 tag works for both boards! :shipit: ❗️

[paulreimer]

Also, incidentally my FT2232H is ready to go at 5Mbaud, but I've never been able to push that fast to the ESP32 (which I thought could do up to 4Mbaud?) 2Mbaud is the fastest I've been able to go. Is this a macOS issue possibly?

[projectgus]

Thanks @paulreimer . ESP-IDF should be updated with v2.2.1 shortly.

[projectgus]

, incidentally my FT2232H is ready to go at 5Mbaud, but I've never been able to push that fast to the ESP32 (which I thought could do up to 4Mbaud?) 2Mbaud is the fastest I've been able to go.

I'm not sure I understand what you're telling me here. There are a few factors for maximum baud rate on ESP32, one of which is board layout. My FTDI-based dev boards flash reliably at 2Mbaud and almost reliably at 4Mbaud. I've been meaning to look into this closer in the future, but even at 2Mbaud most of the flashing time is spent doing things other than serial transfer so I haven't prioritised it too high.

Is this a macOS issue possibly?

The bug fixed in v2.2.1 (2ec0bc0) was reported (privately) and confirmed fixed on macOS, and everyone else reporting in this issue thread appears to be using macOS (although we don't have confirmation v2.2.1 fixes all these issues, I suspect it does.) It seems like macOS flushes some incoming data if the serial port settings are set (even if they're not changed).

[paulreimer]

I guess I'm wondering why there was a macOS-specific regression (related to a high baud rate), and so I thought then, maybe you guys are all using Linux (or Windows)? So maybe that could explain some of my other serial issues I've encountered along the way, such as why I can't hit the baud rates I wanted to. macOS has been fine so far, but if I'm at a disadvantage e.g. from not having a linux box, then I'd like to know! At least it might be closer to your CI environment.

Sorry for the confusion, I don't think it is related to this specific issue. I put those things together in my head and it wasn't really clear.

[projectgus]

Oh, I see now. I happen to use Linux & Windows (and I'm the main esptool.py developer). A number of the ESP-IDF developers use macOS as their main OS, and @igrr (who uses macOS) noticed this regression shortly after we updated esptool.py there. We worked together to get some dtruss system call traces, which is how we figured out the "buffer flush on ioctl" root cause.

So you're not at any significant disadvantage by using macOS. I'm also thinking it may be time I get a Mac Mini or something to run the occasional regression test on.

[marcelstoer]

I finally found the time to confirm that this is fixed with 2.2.1 - at least for me.

[projectgus]

Thanks @marcelstoer :)