search

espressif / arduino-esp32

Arduino core for the ESP32
GNU Lesser General Public License v2.1
12.99k stars 7.29k forks source link

OTA unreliable in v3 #9742

Closed mozgy closed 2 weeks ago

mozgy commented 1 month ago

Board

ESP32 CAM

Device Description

-

Hardware Configuration

-

Version

v3.0.0

IDE Name

PlatformIO

Operating System

Windows 10

Flash frequency

what's in the platformio json files

PSRAM enabled

yes

Upload speed

OTA

Description

OTA times out. If I comment out line webText += String( 161 ); OTA completes fine.
In v2 I can add whatever text to it and OTA always works just fine.

Sketch

void initOTA( void ) {

  ArduinoOTA.setHostname( "OTA-min" );

  ArduinoOTA
    .onStart([]() {
      String type;
      if (ArduinoOTA.getCommand() == U_FLASH)
        type = "sketch";
      else // U_SPIFFS
        type = "filesystem";

      esp_err_t err = esp_camera_deinit();

      Serial.println("Start updating " + type);
    })
    .onEnd([]() {
      Serial.println("\nEnd");
    })
    .onProgress([](unsigned int progress, unsigned int total) {
      Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
    })
    .onError([](ota_error_t error) {
      Serial.printf("Error[%u]: ", error);
      if (error == OTA_AUTH_ERROR) Serial.println("Auth Failed");
      else if (error == OTA_BEGIN_ERROR) Serial.println("Begin Failed");
      else if (error == OTA_CONNECT_ERROR) Serial.println("Connect Failed");
      else if (error == OTA_RECEIVE_ERROR) Serial.println("Receive Failed");
      else if (error == OTA_END_ERROR) Serial.println("End Failed");
    });

  ArduinoOTA.begin();
}

void initWiFi( void ) {
  WiFi.softAPdisconnect( true );
  WiFi.setMinSecurity( WIFI_AUTH_WPA_PSK );
  WiFi.mode( WIFI_STA );
  WiFi.setHostname( "OTA-min" );
  WiFi.begin( ssid, password );
  WiFi.setSleep( false );
}

void asyncHandleRoot( AsyncWebServerRequest *request ) {

  String webText;

  webText = "<!doctype html><html><head><title>OTA Minimum</title></head><body>";
  webText += "OTA Minimum<br>";
  webText += "161";
  webText += String( 161 );
  webText += "</body></html>";

  request->send( 200, "text/html", webText );

}

void asyncHandleNotFound( AsyncWebServerRequest *request ) {
  request->send( 404 );
}

void initAsyncWebServer( void ) {
  asyncWebServer.on( "/", HTTP_GET, asyncHandleRoot );
  asyncWebServer.onNotFound( asyncHandleNotFound );
  asyncWebServer.begin();
}

void setup() {
  Serial.begin( 115200 );
  delay( 400 );

  initWiFi();
  initAsyncWebServer();
  initOTA();
}

void loop() {
  ArduinoOTA.handle();
}

Debug Message

Linking .pio\build\esp32cam\firmware.elf
Retrieving maximum program size .pio\build\esp32cam\firmware.elf
Checking size .pio\build\esp32cam\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  16.1% (used 52632 bytes from 327680 bytes)
Flash: [=======   ]  66.3% (used 1304357 bytes from 1966080 bytes)
Building .pio\build\esp32cam\firmware.bin
esptool.py v4.5.1
Creating esp32 image...
Merged 27 ELF sections
Successfully created esp32 image.
Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = espota
Uploading .pio\build\esp32cam\firmware.bin
00:17:54 [DEBUG]: Options: Namespace(esp_ip='192.168.1.161', host_ip='0.0.0.0', esp_port=3232, host_port=49083, auth='', image='.pio\\build\\esp32cam\\firmware.bin', spiffs=False, debug=True, progress=True, timeout=10)
00:17:54 [INFO]: Starting on 0.0.0.0:49083
00:17:54 [INFO]: Upload size: 1310928
Sending invitation to 192.168.1.161
00:17:54 [INFO]: Waiting for device...
00:18:04 [ERROR]: No response from device
*** [upload] Error 1
================================================================================== [FAILED] Took 98.73 seconds ==================================================================================

Environment    Status    Duration
-------------  --------  ------------
esp32cam       FAILED    00:01:38.730

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

mozgy commented 1 month ago

A bit more perspective - compiling with

platform_packages =
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.17
  ; framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.0
  ; framework-arduinoespressif32-libs @ https://github.com/espressif/arduino-esp32/releases/download/3.0.0/esp32-arduino-libs-3.0.0.zip

OTA upload works always, regardless if OTA server (aka board receiving fw) is v2 or v3, if compiling with

platform_packages =
  ; framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2.0.17
  framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#3.0.0
  framework-arduinoespressif32-libs @ https://github.com/espressif/arduino-esp32/releases/download/3.0.0/esp32-arduino-libs-3.0.0.zip

OTA upload works only if I comment out that above mentioned line, regardless what version is OTA server.

me-no-dev commented 1 month ago

And if you remove the web server all together?

mozgy commented 1 month ago

Web server is there so I can definitely be sure what fw version is active. I'll try later with even web server stripped down.

More testing ->

PS C:\Users\mozgy\.platformio\packages\framework-arduinoespressif32\tools> .\espota.exe --ip 192.168.1.161 --file=C:\fw-ok.bin --debug --progress
11:57:56 [DEBUG]: Options: Namespace(auth='', debug=True, esp_ip='192.168.1.161', esp_port=3232, host_ip='0.0.0.0', host_port=13591, image='C:\\fw-ok.bin', progress=True, spiffs=False, timeout=10)
11:57:56 [INFO]: Starting on 0.0.0.0:13591
11:57:56 [INFO]: Upload size: 1310688
Sending invitation to 192.168.1.161
11:57:56 [INFO]: Waiting for device...
Uploading: [============================================================] 100% Done...

11:58:10 [INFO]: Waiting for result...
11:58:10 [INFO]: Result: OK
11:58:10 [INFO]: Success

PS C:\Users\mozgy\.platformio\packages\framework-arduinoespressif32\tools> .\espota.exe --ip 192.168.1.161 --file=C:\firmware.bin --debug --progress
12:00:40 [DEBUG]: Options: Namespace(auth='', debug=True, esp_ip='192.168.1.161', esp_port=3232, host_ip='0.0.0.0', host_port=56759, image='C:\\firmware.bin', progress=True, spiffs=False, timeout=10)
12:00:40 [INFO]: Starting on 0.0.0.0:56759
12:00:40 [INFO]: Upload size: 1311120
Sending invitation to 192.168.1.161
12:00:40 [INFO]: Waiting for device...
12:00:50 [ERROR]: No response from device

PS C:\Users\mozgy\.platformio\packages\framework-arduinoespressif32\tools> .\espota.exe --ip 192.168.1.161 --file=C:\fw-ok.bin --debug --progress
12:02:00 [DEBUG]: Options: Namespace(auth='', debug=True, esp_ip='192.168.1.161', esp_port=3232, host_ip='0.0.0.0', host_port=26967, image='C:\\fw-ok.bin', progress=True, spiffs=False, timeout=10)
12:02:00 [INFO]: Starting on 0.0.0.0:26967
12:02:00 [INFO]: Upload size: 1310688
Sending invitation to 192.168.1.161
12:02:00 [INFO]: Waiting for device...
Uploading: [============================================================] 100% Done...

12:02:14 [INFO]: Waiting for result...
12:02:14 [INFO]: Result: OK
12:02:14 [INFO]: Success
mozgy commented 1 month ago

Still the same timeout even with no webserver compiled in -

PS C:\Users\mozgy\.platformio\packages\framework-arduinoespressif32\tools> .\espota.exe --ip 192.168.1.161 --port=3232 --file=C:\OTA\firmware.bin --debug --progress
12:30:25 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.1.161', 'host_port': 29531, 'image': 'C:\\OTA\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': '', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
12:30:25 [INFO]: Starting on 0.0.0.0:29531
12:30:25 [INFO]: Upload size: 811168
Sending invitation to 192.168.1.161
12:30:25 [INFO]: Waiting for device...
Uploading: [============================================================] 100% Done...

12:30:34 [INFO]: Waiting for result...
12:30:34 [INFO]: Result: OK
12:30:34 [INFO]: Success

$ ping 192.168.1.161
PING 192.168.1.161 (192.168.1.161) 56(84) bytes of data.
64 bytes from 192.168.1.161: icmp_seq=1 ttl=63 time=2.85 ms
$ telnet 192.168.1.161 8080
Trying 192.168.1.161...
telnet: Unable to connect to remote host: Connection refused

PS C:\Users\mozgy\.platformio\packages\framework-arduinoespressif32\tools> .\espota.exe --ip 192.168.1.161 --port=3232 --file=C:\OTA\fw-bad.bin --debug --progress
12:31:33 [DEBUG]: Options: {'timeout': 10, 'esp_ip': '192.168.1.161', 'host_port': 15468, 'image': 'C:\\OTA\\fw-bad.bin', 'host_ip': '0.0.0.0', 'auth': '', 'esp_port': 3232, 'spiffs': False, 'debug': True, 'progress': True}
12:31:33 [INFO]: Starting on 0.0.0.0:15468
12:31:33 [INFO]: Upload size: 1311120
Sending invitation to 192.168.1.161
12:31:33 [INFO]: Waiting for device...
12:31:43 [ERROR]: No response from device
mozgy commented 1 month ago

Redid all tests on Linux Fedora box and the results were the same so it's not OS related. Here are tcpdumps of both attempt results -

367 50.110710   192.168.1.44    192.168.1.161   UDP 91  52753 → 3232 Len=49
368 50.116919   192.168.1.161   192.168.1.44    UDP 60  3232 → 52753 Len=2
369 50.127483   192.168.1.161   192.168.1.44    TCP 60  56439 → 58748 [SYN] Seq=0 Win=5760 Len=0 MSS=1436
370 50.127612   192.168.1.44    192.168.1.161   TCP 58  58748 → 56439 [SYN, ACK] Seq=0 Ack=1 Win=64620 Len=0 MSS=1460
371 50.130164   192.168.1.161   192.168.1.44    TCP 60  56439 → 58748 [ACK] Seq=1 Ack=1 Win=5760 Len=0
...

73  12.300023   192.168.1.44    192.168.1.161   UDP 91  52305 → 3232 Len=49
76  12.306779   192.168.1.161   192.168.1.44    UDP 60  3232 → 52305 Len=2
me-no-dev commented 1 month ago

so when it failed, the ESP did respond, just the ESP did not initiate the pull? Hmm... maybe some more debug is needed

mozgy commented 4 weeks ago

more debug ->

[  8460][D][Updater.cpp:140] begin(): OTA Partition: app0
[  8465][E][Updater.cpp:162] begin(): too large 1355024 > 1310720
[  8471][E][ArduinoOTA.cpp:242] _runUpdate(): Begin ERROR: Bad Size Given
Error[1]: Begin Failed

-a----         2.6.2024.     12:00        1311120 fw-bad.bin
-a----         2.6.2024.     11:56        1310688 fw-ok.bin

interesting is that both FW flash fine with esptool

TD-er commented 3 weeks ago

I also noticed some issues OTA updating some nodes. These are updated via the web interface (standard webserver) thus via a POST call.

Mainly for ESP32-nodes with larger flash, which do run larger build files.

Had not yet looked into this, just noticed by me and a fellow developer of my project (ESPEasy) that it was (quite) a bit harder to flash some nodes via OTA updates.

If an update failed, you need to powercycle these units (they don't have a reset button, so not sure if that would have worked).

VojtechBartoska commented 2 weeks ago

Up to our investigation, we did not see any significant change.

So far, we are not able to reproduce this and it is very specific to your usecase.

We discussed this ticket within the team and did not find any action items here due to that I am closing this ticket.

Thanks for understanding.

mozgy commented 2 weeks ago

No worries, in the end my issue is that OTA cannot change partition format ..