esptool fails to write to ESP8266 with PUYA flash chip - The chip stopped responding (ESPTOOL-699)

wibbit commented 1 year ago

Operating System

Fedora38

Esptool Version

esptool-4.5.1-2.fc38.noarch

Python Version

https://packages.icinga.com/epel/8/release

Chip Description

ESP8266

Device Description

Wemos D1 Mini v4

Hardware Configuration

No response

How is Esptool Run

CLI

Full Esptool Command Line that Was Run

esptool write_flash 0x0 OTGW-firmware-0.10.2+50c3ed2.ino.bin 0x200000 OTGW-firmware-0.10.2+50c3ed2.littlefs.bin

Esptool Output

otgw]$ esptool write_flash 0x0 OTGW-firmware-0.10.2+50c3ed2.ino.bin 0x200000 OTGW-firmware-0.10.2+50c3ed2.littlefs.bin
esptool.py v4.5.1
Found 2 serial ports
Serial port /dev/ttyUSB1
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 7c:87:ce:be:25:e0
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00000000 to 0x00091fff...
Flash will be erased from 0x00200000 to 0x002f9fff...
Compressed 597664 bytes to 398357...
Wrote 597664 bytes (398357 compressed) at 0x00000000 in 48.1 seconds (effective 99.4 kbit/s)...
Hash of data verified.
Compressed 1024000 bytes to 75348...
Writing at 0x0025e8e9... (40 %)Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/esptool/__init__.py", line 1032, in _main
    main()
  File "/usr/lib/python3.11/site-packages/esptool/__init__.py", line 832, in main
    operation_func(esp, args)
  File "/usr/lib/python3.11/site-packages/esptool/cmds.py", line 586, in write_flash
    esp.flash_defl_block(block, seq, timeout=timeout)
  File "/usr/lib/python3.11/site-packages/esptool/loader.py", line 131, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/esptool/loader.py", line 1030, in flash_defl_block
    self.check_command(
  File "/usr/lib/python3.11/site-packages/esptool/loader.py", line 435, in check_command
    val, data = self.command(op, data, chk, timeout=timeout)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/esptool/loader.py", line 404, in command
    p = self.read()
        ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/esptool/loader.py", line 337, in read
    return next(self._slip_reader)
           ^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

A fatal error occurred: The chip stopped responding.

More Information

I am able to always successfully flash the ino file, however, flashing the littlefs always failed.

I tried pretty much all baud rates from 9600 up to 115200, the errors would sometimes vary, including stating an "A serial exception error occurred: Write timeout".

This was tested with 2 v4 cards, multiple USB cables, multiple ports, and multiple machines.

I eventually got this to work (100% of the time), when including the --no-compress flag to the write_flash subcommand.

I've spoken to someone who's not had an issue writing to this hardware, however, they are using esptool v2.8

Though I've got this working, there was no apparent output to suggest --no-compress could resolve the issue, and as a new user this was a very painful experience.

Posting here so that. a) the tooling can some how be updated to address this failure scenario. b) some one searching for the same issue, may find this bug report, and the solution that worked for me.

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

[X] I confirm I have read the troubleshooting guide.

dobairoland commented 1 year ago

Hi @wibbit. Are you able to flash it if you add --no-compress after write_flash?

EDIT: I sorry, for some reason I haven't read everything.

radimkarnis commented 1 year ago

Hi @wibbit,

I have tried reproducing this with several ESP8266's, esptool v4.5.1, and these bin files (please correct me if these binaries are not the right ones).

I was not able to reproduce the failure:

esptool.py write_flash 0x0 OTGW-firmware-0.10.2+50c3ed2.ino.bin 0x200000 OTGW-firmware-0.10.2+50c3ed2.littlefs.bin

esptool.py v4.5.1
Found 5 serial ports
Serial port /dev/cu.wlan-debug
/dev/cu.wlan-debug failed to connect: Could not open /dev/cu.wlan-debug, the port doesn't exist
Serial port /dev/cu.usbserial-0001
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: c8:2b:96:30:29:33
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Flash will be erased from 0x00000000 to 0x00091fff...
Flash will be erased from 0x00200000 to 0x002f9fff...
Compressed 597664 bytes to 398357...
Wrote 597664 bytes (398357 compressed) at 0x00000000 in 35.3 seconds (effective 135.5 kbit/s)...
Hash of data verified.
Compressed 1024000 bytes to 75348...
Wrote 1024000 bytes (75348 compressed) at 0x00200000 in 8.1 seconds (effective 1011.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Could you please try the following:

Updating esptool to v4.6.2
Flashing another ESP8266 (not a Wemos D1 Mini v4, ideally another devkit with a different kind of a USB-to-Serial bridge). I understand this might not be an option for you

radimkarnis commented 1 year ago

Two more things to try:

Downgrading esptool to v2.8
Custom configuration with a config file:

You can create an esptool.cfg file in your current directory with this content:

[esptool]
timeout = 3
max_timeout = 240
erase_write_timeout_per_mb = 40
mem_end_rom_timeout = 0.2
serial_write_timeout = 10

This custom configuration will then get loaded by esptool and will allow you to tune it for your environment. Can you please try increasing some of the timeouts and see if that helps?

wibbit commented 1 year ago

I've not yet upgraded/downgraded esptool, however increasing the timeout from 3 to 30 allowed the process to finish without including --no-compress.

I'd increased in increments of 5, with 25 still failing.

I don't have another non-wemos based ESP8266 board to try with, but I can certainly go ahead and try out the other versions, probably Wednesday evening.

radimkarnis commented 1 year ago

Thanks for the investigation! Having to increase the timeout to 30s is a warning sign. None of the operations should take that much time. There might be an issue with the drivers of the ch340c USB-to-UART chip. Please let me know if other devkits work for you!

wibbit commented 1 year ago

Working with some people on a Discord channel, we think we have narrowed this down to the flash chip that is present.

The v3's (and good v4's) tested come with a T25S32 chip whereas the problematic v4's come with a puya branded chip, indicative throughput differences below. Wrote 1032192 bytes at 0x00200000 in 62.8 seconds (131.5 kbit/s)... v's Wrote 1032192 bytes at 0x00200000 in 23.0 seconds (359.1 kbit/s)...

So, I assume the problem is that the CPU on the card is waiting on IO, and esptool eventually times out.

This is obviously an issue with the card, I'm not sure if there is anything in esptool that could be done to notify/warn a user, I'll leave this down to you lot to decide, as you obviously have a FAR more intimate understanding of the ecosystem then I do.

radimkarnis commented 1 year ago

Nice investigation! Could you please run esptool.py flash_id on a faulty and a working unit and post the result?

Maybe we could detect the faulty flash chip in esptool and at least provide a warning or a note. Thank you!

wibbit commented 1 year ago

Good chip Manufacturer: 5e Device: 4016 Detected flash size: 4MB

Problematic chip Manufacturer: 85 Device: 6016 Detected flash size: 4MB

TD-er commented 11 months ago

OMG... I really hoped those awfull Puya chips would be a one-time only disaster with their 1M chips. But apparently they now also have 4M chips.

Do you know if those 4M units also need the 'PUYA' patch for ESP8266, which essentially does read the entire sector, alter it in memory and write it back, because those chips actually write what you tell them to write, instead of only turning a 1 into a 0 like normal flash chips do.

radimkarnis commented 11 months ago

@wibbit I will try to get my hands on one of these devkits with Puya flash chips and investigate. Thanks for your time! I will keep this open until then.

radimkarnis commented 11 months ago

So I have ordered a batch of devkits, that were supposed to have the problematic PUYA flash chips. Unfortunately, they all arrived with the good stuff - Manufacturer: 5e, Device: 4016.

I guess the manufacturer knows about the issues and maybe started sourcing only the good flash chips.

wibbit commented 11 months ago

Well that's not infuriating at all, i wonder if I can get a refund and a replacement product sent out.

Is there any thing I can do to provide additional information?

Sbsin-Ks commented 9 months ago

Hi there Thank you so much for this hint. I stuck at the same problem and also tried every baud rate, different com ports and PCs. The option --no-compress made it work. I have uploaded the files with the actual esptool version v4.6.2.

@radimkarnis I have ordered multiple v4 boards with the problematic chip and I would like to conribute one chip to you if it helps others not wasting hours of playing try and error.

radimkarnis commented 8 months ago

@Sbsin-Ks Thanks for the heads up and the --no-compress workaround!

I would like to conribute one chip to you if it helps others not wasting hours of playing try and error.

That would be appreciated, I didn't manage to get my hands on one of these. On the other hand, we didn't get any more reports of issues with these chips in the wild. Do you reckon adding a warning to esptool would be useful?

espressif / esptool