Esptool tool always fail at the connecting stage but AT+GMR command returns chip data (ESPTOOL-844)

dmigwi commented 7 months ago

Operating System

macOS 12.7.4

Esptool Version

v3.1.2

Python Version

Python 3.9.4

Chip Description

ESP-01s

Device Description

Leonardo Board as serial to TTL converter: 2017-Rev3d. On the Leonardo board I have built a custom esp programming board based on designs available here. Full page article can be found here. https://tttapa.github.io/ESP8266/Chap02%20-%20Hardware.html

Hardware Configuration

I am using serial communication between Leonardo board and ESP-01s chip. Leonardo --- ESP-01s Rx --- Tx Tx --- Rx (Voltage divider resistors are used to supply 3.3v to the Rx ESP pin)

I used AMS1117-3.3 LDO to guarantee a consistent output of 3.3v to ESP-01 power pins

How is Esptool Run

Arduino IDE 2.3.2

Full Esptool Command Line that Was Run

No response

Esptool Output

Output when I try to program the ESP module:

. Variables and constants in RAM (global, static), used 28104 / 80192 bytes (35%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ DATA     1496     initialized variables
╠══ RODATA   920      constants       
╚══ BSS      25688    zeroed variables
. Instruction RAM (IRAM_ATTR, ICACHE_RAM_ATTR), used 59667 / 65536 bytes (91%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ ICACHE   32768    reserved space for flash instruction cache
╚══ IRAM     26899    code in IRAM    
. Code in flash (default, ICACHE_FLASH_ATTR), used 232148 / 1048576 bytes (22%)
║   SEGMENT  BYTES    DESCRIPTION
╚══ IROM     232148   code in flash   
esptool.py v3.0
Serial port /dev/cu.usbmodem141401
Connecting........_____....._____....._____....._____....._____....._____....._____
A fatal esptool.py error occurred: Failed to connect to ESP8266: Timed out waiting for packet header

Output When I run AT commands:

```Log
AT+GMR

AT version:1.2.0.0(Jul  1 2016 20:04:45)
SDK version:1.5.4.1(39cb9a32)
Ai-Thinker Technology Co. Ltd.
Dec  2 2016 14:21:16
OK


### More Information

Because of the lack of a dedicated chip in Leonardo to manage serial communication, I am using a serial passthrough code running on a the Arduino board to communicate with the ESP-01 module.

```C++
void setup() {
  Serial.begin(115200);
  Serial1.begin(115200);
}

void loop() {
  if (Serial.available()) {      // If anything comes in Serial (USB),
    int data = Serial.read();
    Serial1.write(data);   // read it and send it out Serial1 (pins 0 & 1)
  }

  if (Serial1.available()) {     // If anything comes in Serial1 (pins 0 & 1)
    int data = Serial1.read();
    Serial.write(data);   // read it and send it out Serial (USB)
  }
}

Other Steps to Reproduce

In the last 2 weeks, I have gone through countless tutorials and documentations about ESP-01 programming but not even once did I manage to get past the connection stage when attempting to program with Arduino IDE. The AT commands work so well, I wonder how exactly programming the Arduino can always fail at the connection stage with the same error always: A fatal esptool.py error occurred: Failed to connect to ESP8266: Timed out waiting for packet header

I Have Read the Troubleshooting Guide

[X] I confirm I have read the troubleshooting guide.

radimkarnis commented 7 months ago

Hi @dmigwi, I can't help you with anything AT-related. The only relevant error here is the esptool.py error: Timed out waiting for packet header. That means the serial communication is not working or your auto reset circuit is not working. Since we see the AT console output, we know that serial comms are fine. Therefore, you need to focus on resetting the chip into the bootloader.

If your circuit doesn't support this, you need to do it manually - pull GPIO0 low and reset the chip. Then try flashing (you can keep pulling GPIO0 low during the process).

You can get more tips in the troubleshooting guide.

dmigwi commented 7 months ago

Hi @radimkarnis Thanks for the feedback above. I believe I have been able to access the chip flashing mode by shorting the GP00 pin to GND before powering on the device. Using the serial passthrough code I am able to record communication to the ESP-01s chip but not from it as shown in the clip below.

https://github.com/espressif/esptool/assets/22055953/4bc72670-8799-4078-a73b-0c52df576037

In this json, https://arduino.esp8266.com/stable/package_esp8266com_index.json your stable releases only go up v3.1.2 whereas you've shipped binaries for v4.x.x with configuration file support. Could you help me get the updated json that I can use the latest binaries to test with?

Maybe some few changes on DTS and RTS pins configurations will fix my issue. I have noticed that Leonardo board doesn't recommend reseting the board as it only has one CPU that manages the both serial communication and executing the arduino code. This is different from what other boards like Arduino UNO has.

dmigwi commented 7 months ago

Today is the first time I have seen a different error message. It is also the first time I have gone past the connection issues I have been experiencing for the past couple of days. This happened after I have identified two bugs currently in the code base since v2.4.1 till now.

flushInput() and flushOutput() were deprecated by pyserial close to two years ago but the are still used in the code base. https://github.com/espressif/esptool/blob/3d55c74efdabb7e9f542914b7f214ed0d0b205b2/esptool/loader.py#L506 And https://github.com/espressif/esptool/blob/3d55c74efdabb7e9f542914b7f214ed0d0b205b2/esptool/loader.py#L593

replacing those two with the following got me here.

self._port.reset_input_buffer()

self._port.reset_output_buffer()

dmigwi commented 7 months ago

After weeks on trying, finally I have managed to get the blinking sketch running on ESP-01 WIFI module. Bugs resolved on v3.1.2 esptool to get this much progress.

Updated the deprecated pyserial methods: flushInput() and flushOutput()
On Leonardo Arduino board, DTR must be set to True for the serial connection between the Arduino and ESP-01s module to be detected via the Arduino IDE. Pyserial does this by default thus self._connect_attempt(mode='no_reset', esp32r0_delay=False) should be used otherwise connection will never be established. Reference: https://forum.arduino.cc/t/leonardo-wont-initiate-serial-communication/117072/9
When the available buffer is exceeded while sending data, the following error happens:
```
A fatal esptool.py error occurred: Write timeout
```
on reducing the size of block data sent further, this new error happens.
```
a fatal esptool.py error occurred: Timed out waiting for packet header
```

The solution that worked for me is to reduce it further and my configuration that worked is:

    # Maximum block sized for RAM and Flash writes, respectively.
    ESP_RAM_BLOCK = 0x140

    FLASH_WRITE_SIZE = 0x140

NB: FLASH_WRITE_SIZE is declared severally(a very bad design) at least in v3.1.2 so I had to set it on multiple places to get it working.

radimkarnis commented 7 months ago

Hello @dmigwi, thank you for your investigation and reports.I am glad you got it working.

We do not actively develop v3 anymore (it receives critical bugfixes, though). Your issue seems to be very specific - having to reduce block sizes usually points at buggy drivers or a noisy environment.

I'll close this issue now. If you feel like some of this should be addressed, please try with the latest esptool (it doesn't have to run in the Arduino framework, standalone will do) and open a new issue.

dmigwi commented 7 months ago

@radimkarnis What would it take to update https://arduino.esp8266.com/stable/package_esp8266com_index.json to have the latest releases supported. v3.1.2 is too old where as there is v4.x.x available? Also the pyserial deprecated methods are still in the latest release. Need help addressing that or you've got it under control?

radimkarnis commented 7 months ago

@dmigwi few things to point out:

you mention v3.1.2, but that is not esptool.py version, but the ESP8266 Arduino core version
The core repository imports esptool as a git submodule, you can see it uses the v3.0 release commit here: https://github.com/esp8266/Arduino/tree/master/tools
You can try updating that git submodule to a more recent version but starting from v4.0 compatibility is not ensured, there might be breaking changes

Also the pyserial deprecated methods are still in the latest release. Need help addressing that or you've got it under control?

We didn't feel the need to update these, as these are just aliases for the new interface. If you feel like it, you can always send a pull request, we welcome every contribution.

espressif / esptool