search

Links2004 / arduinoWebSockets

arduinoWebSockets
GNU Lesser General Public License v2.1
1.89k stars 555 forks source link

OTA over websocket #357

Closed Hokyjack closed 2 years ago

Hokyjack commented 6 years ago

Hello, do you think it would be possible to make OTA update with usage of websockets? I know there are libraries to do OTA via http server, or via UDP socket, but I haven't see any attemt to OTA over WS.

I was thinking about writting my own function, but I am lost about how to handle the binary file on the ESP side. Should you first send whole firmware as a BIN and save it and then update, or do OTA "on fly" when streaming the data via websocket?

Thank you.

Links2004 commented 6 years ago

its not ready to go but It is possible to program. and not that hard if your server sends the program via Websocket fragments (all in one will not fit the RAM)

the basic ESP8266 update process will be done in multible steps:

  1. download the new program to free flash
  2. check if all data is written there correct
  3. reboot
  4. the boot loader will copy the new program over the old
  5. the new Programm is started

check: https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Updater.cpp https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Updater.h

you can easy find some examples on how to use in the multible OTA implemntations

https://github.com/esp8266/Arduino/blob/fb7c5198561b88d3b01914ea35d7c42e3599386f/libraries/ArduinoOTA/ArduinoOTA.cpp#L309 https://github.com/esp8266/Arduino/blob/61cd8d83859524db0066a647de3de3f6a0039bb2/libraries/ESP8266WebServer/examples/WebUpdate/WebUpdate.ino#L44 https://github.com/esp8266/Arduino/blob/b81ef01ef769beb912f78458e6ba04b0c446e76d/libraries/ESP8266HTTPUpdateServer/src/ESP8266HTTPUpdateServer.cpp#L80

Hokyjack commented 6 years ago

@Links2004 Thank you for your advices. I was able to to send whole firmware via 8kB chunks over WS and save it into file to SPIFFS.

However, I am not sure how to make UpdateClass take the file and do the actual OTA.

I tried this, but it just hangs on for few seconds and then do nothing.

File & WebsocketClient::openFile(const char * path, const char * mode) {
    File file = SPIFFS.open(String("/") + String(path), mode);
}

...

File & file = client->openFile("firmware", "r");
Serial.println("Try to update...");
Update.writeStream(file);

Or should I use write(uint8_t *data, size_t len) ? But I am not sure about casting the char * data to uint8_t... I am not a C++ pro, so there might be something I am doing badly. Any ideas please?

Links2004 commented 6 years ago

you dont need to store the FW in the SPIFFS, you can direct user the write of the UpdateClass command on every callback call in the websocket. but keep in mind that the UpdateClass need to be configured correctly (begin, setMD5) for a update.

more or less:

  1. Update.begin
  2. Update.setMD5
  3. Update.write until all data is there (multible calls with the 8K blocks - check Update.isFinished)
  4. Update.end
  5. ESP.restart

each step needs needs error handling (check the examples linked above)

Hokyjack commented 6 years ago

I GOT IT WORKING!!, I love you @Links2004 for your help and directions! I can post here some code sample once I get it cleaned a bit.

Few issues I ran into, sometimes when sending the chunks device reconnects to the WS server. Waiting to send another chunk by sending confirm message after each and setting hard 150 ms delays helps a bit, but still can happen sometime.

Update failed at the end, printing error message told me MD5 hash was wrong, I tried send it in lowercase and voila, problem solved.

Links2004 commented 6 years ago

you are welcome. the flash writing need some time, during this time the the WiFi buffer can be overloaded when you send to much data ;) may enabling the debug messages gives more info about the disconnect.

Hokyjack commented 6 years ago

hi, I turned on debug messages and this come out when failing last time update:

...
[WS][0][handleWebsocketWaitFor] size: 2 cWsRXsize: 0
[readCb] n: 2 t: 63190
[WS][0][handleWebsocketWaitFor][readCb] size: 2 ok: 1
[WS][0][handleWebsocketWaitFor] size: 4 cWsRXsize: 2
[readCb] n: 2 t: 63201
[WS][0][handleWebsocketWaitFor][readCb] size: 4 ok: 1
[WS][0][handleWebsocket] ------- read massage frame -------
[WS][0][handleWebsocket] fin: 1 rsv1: 0 rsv2: 0 rsv3 0  opCode: 2
[WS][0][handleWebsocket] mask: 0 payloadLen: 8192
[readCb] n: 8192 t: 63223
[WSc] Recieved binary length: 8192
OTA recieving 8192 bytes
[WS][0][sendFrame] ------- send message frame -------
[WS][0][sendFrame] fin: 1 opCode: 1 mask: 1 length: 12 headerToPayload: 0
[WS][0][sendFrame] text: OTA CHUNK OK
[WS][0][sendFrame] pack to one TCP package...
[write] n: 18 t: 63341
[WS][0][sendFrame] sending Frame Done (3225us).
[WS][0][handleWebsocketWaitFor] size: 2 cWsRXsize: 0
[readCb] n: 2 t: 63352
[WS][0][handleWebsocketWaitFor][readCb] size: 2 ok: 1
[WS][0][handleWebsocket] ------- read massage frame -------
[WS][0][handleWebsocket] fin: 1 rsv1: 0 rsv2: 0 rsv3 0  opCode: 9
[WS][0][handleWebsocket] mask: 0 payloadLen: 0
[WS][0][sendFrame] ------- send message frame -------
[WS][0][sendFrame] fin: 1 opCode: 10 mask: 1 length: 0 headerToPayload: 0
[write] n: 6 t: 63385
[WS][0][sendFrame] sending Frame Done (2588us).
[WS][0][handleWebsocketWaitFor] size: 2 cWsRXsize: 0
[readCb] n: 2 t: 63396
[WS][0][handleWebsocketWaitFor][readCb] size: 2 ok: 1
[WS][0][handleWebsocketWaitFor] size: 4 cWsRXsize: 2
[readCb] n: 2 t: 63407
[WS][0][handleWebsocketWaitFor][readCb] size: 4 ok: 1
[WS][0][handleWebsocket] ------- read massage frame -------
[WS][0][handleWebsocket] fin: 1 rsv1: 0 rsv2: 0 rsv3 0  opCode: 2
[WS][0][handleWebsocket] mask: 0 payloadLen: 8192
[readCb] n: 8192 t: 63429
[readCb] receive TIMEOUT! 2001
[WS][0][handleWebsocket] missing data!
[WS][0][handleWebsocket] clientDisconnect code: 1002
[WS][0][sendFrame] ------- send message frame -------
[WS][0][sendFrame] fin: 1 opCode: 8 mask: 0 length: 2 headerToPayload: 0
[WS][0][sendFrame] pack to one TCP package...
[write] n: 4 t: 65447
[WS][0][sendFrame] sending Frame Done (2616us).
[WS-Client] client disconnected.
[WSc] Disconnected!
[WS-Client] connect ws...
[WS-Client] connected to 192.168.1.7:8887.
[WS-Client][sendHeader] sending header...
[WS-Client][sendHeader] handshake GET / HTTP/1.1
Host: 192.168.1.7:8887
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: FBP61PjQKW9ZoFbH8IMs/Q==
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client
Authorization: 24:0A:C4:8E:7F:E8

[write] n: 277 t: 65535
[WS-Client][sendHeader] sending header... Done (26141us).
[WS-Client][handleHeader] RX: HTTP/1.1 101 Web Socket Protocol Handshake
[WS-Client][handleHeader] RX: Connection: Upgrade
[WS-Client][handleHeader] RX: Date: Sun, 19 Aug 2018 08:40:41 GMT+00:00
[WS-Client][handleHeader] RX: Sec-WebSocket-Accept: Ehbfw5slO+7M9sFoLTC3Wu1g9L0=
[WS-Client][handleHeader] RX: Server: TooTallNate Java-WebSocket
[WS-Client][handleHeader] RX: Upgrade: websocket
[WS-Client][handleHeader] Header read fin.
[WS-Client][handleHeader] Client settings:
[WS-Client][handleHeader]  - cURL: /
[WS-Client][handleHeader]  - cKey: FBP61PjQKW9ZoFbH8IMs/Q==
[WS-Client][handleHeader] Server header:
[WS-Client][handleHeader]  - cCode: 101
[WS-Client][handleHeader]  - cIsUpgrade: 1
[WS-Client][handleHeader]  - cIsWebsocket: 1
[WS-Client][handleHeader]  - cAccept: Ehbfw5slO+7M9sFoLTC3Wu1g9L0=
[WS-Client][handleHeader]  - cProtocol: arduino
[WS-Client][handleHeader]  - cExtensions:
[WS-Client][handleHeader]  - cVersion: 0
[WS-Client][handleHeader]  - cSessionId:
[WS-Client][handleHeader] Websocket connection init done.
[WS][0][headerDone] Header Handling Done.
[WSc] Connected to url: /
ERROR: when OTA updating

I suppose there is an issue with ping-pong communication during heavy load, so the server thing the device timed-out? I use https://github.com/TooTallNate/Java-WebSocket on the Android side, with server.setConnectionLostTimeout( 40 ); which is a lot I think. Could this be the problem?

Hokyjack commented 6 years ago

I tried to disable setConnectionLostTimeout completely (0), but still the same issue.

Links2004 commented 6 years ago

the log shows that the WS code does not get all the needed data. may its lost in the ESP TCP stack may the TCP stack of the ESP can not store the 8K at once all the time. try to lower the 8K blocks to 4K.

[readCb] n: 8192 t: 63429
[readCb] receive TIMEOUT! 2001
[WS][0][handleWebsocket] missing data!

ping / pong is normally only done when there is no data flow, so as long there is a data send you will not see any ping / pong. since seeing data shows already that the other side is alive ;)

Hokyjack commented 6 years ago

Lowering to 4K blocks failed just before the end (same issue), but setting to 2K chunks went OK as I just tried 3 times in a row. It's great, But still I am wondering why 8K chunks sometimes failed and sometimes not. It is way slower (3x-4x) with 2K chunks. Since I acknowledge every recieved packed on client side, maybe the fix would be to send a RETRY message, instead of dropping and reconnecting to WS, and then just resend the last chunk from server. Sounds like its doable?

Links2004 commented 6 years ago

the WS protocol is brocken when data is missing since it need to be handled as a steam. if you miss a part in the mid you can not decode any data that comes after the missing part. there is no way of recovering after the missing data. for finding the chunk size limiting factor you need to dive in to the ESP TCP layer and after this in to lwip.

the Websocket protocol supports chunking itself via Fragments, if your server supports it you not need to send a ACK back for every chunk.

the callback can handle:

    case WStype_FRAGMENT_BIN_START:
    case WStype_FRAGMENT:
    case WStype_FRAGMENT_FIN:

but your sever needs to send it in fragments for this

bhcuong2008 commented 5 years ago

I also faces this issue when doing OTA via this websocket library.

File tested with ~1MB, and ~31MB.

So I think that there are some critical changes in Wifi / Network stack between those versions. I have not found out solutions yet.

arduino-nour commented 5 years ago

Thank you for your comment I hope that if this problem is solved I will be informed

‫في الاثنين، 26 أغسطس 2019 في 3:30 م تمت كتابة ما يلي بواسطة ‪bhcuong2008‬‏ ‪notifications@github.com‬‏:‬

I also faces this issue when doing OTA via this websocket library.

-

With ESP Arduino library under 1.0.0 ~ 1.0.0-rc4, I could send a lot of frames with chunksize 16KB, each chunk per 5ms via Ethernet. I've not yet tested with Wifi.

But now with 1.0.1 up to the latest 1.0.3.-rc2, if I send a lot of frame, even 1frame/second, still failed.

  • For Ethernet, 4KB frame size is always success. But 8KB, 16KB failed.
  • For Wifi, 1KB frame size also failed. 2KB up, it's always failed. Failed means that Websocket closed connection due to opcode (close code often 1002, 1006), that corrupted data from wificlient data.

File tested with ~1MB, and ~31MB.

So I think that there are some critical changes in Wifi / Network stack between those versions. I have not found out solutions yet.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Links2004/arduinoWebSockets/issues/357?email_source=notifications&email_token=AM6BEE5DGOOXZT4U7SYUGL3QGPEGTA5CNFSM4FPTYHBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EHCAA#issuecomment-524841216, or mute the thread https://github.com/notifications/unsubscribe-auth/AM6BEE7P4NFINSWFLGOQRB3QGPEGTANCNFSM4FPTYHBA .

mark-hahn commented 4 years ago

I can post here some code sample once I get it cleaned a bit.

@Hokyjack: I need this exact thing. Did you ever post your code?

GowthamGottimukkala commented 4 years ago

@Hokyjack I want the same thing. Can you please post the code?

ildarmustafin commented 3 years ago

Waiting

ildarmustafin commented 3 years ago

Waiting

ildarmustafin commented 3 years ago

Waiting

mtsmtsmts commented 3 years ago

waiting too

Tony763 commented 3 years ago

Hi @mtsmtsmts, @ildarmustafin, @GowthamGottimukkala, @mark-hahn, if you are still interested in OTA over Websocket, would You mind testing PR#719?

mark-hahn commented 3 years ago

It's on my todo list, but it may be up to a week before I get the time.

On Wed, Oct 27, 2021 at 1:51 PM Antonín Skala @.***> wrote:

Hi @mtsmtsmts https://github.com/mtsmtsmts, @ildarmustafin https://github.com/ildarmustafin, @GowthamGottimukkala https://github.com/GowthamGottimukkala, @mark-hahn https://github.com/mark-hahn, if you are still interested in OTA over Websocket, would You mind testing PR#719?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Links2004/arduinoWebSockets/issues/357#issuecomment-953300976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGGDPZSZUNXKI7CBPTHB23UJBX6DANCNFSM4FPTYHBA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

MythicalForce commented 2 years ago

is there any code for this?

Tony763 commented 2 years ago

Hi https://github.com/Links2004/arduinoWebSockets/tree/master/examples/esp8266/WebSocketClientOTA

dr3adx commented 2 years ago

@Tony763 bro im getting 


Traceback (most recent call last):
  File "server.py", line 93, in run
    asyncio.run(self.start_)
  File "/usr/lib/python3.8/asyncio/runners.py", line 37, in run
    raise ValueError("a coroutine was expected, got {!r}".format(main))
ValueError: a coroutine was expected, got <bound method WsOtaHandler.start_ of <WsOtaHandler(thread_ota, started daemon 140227056555776)>>
{"type":"greetings","mac":"0C:B8:15:C3:92:08","ip":"192.168.1.200","version":"1.0.0","name":"chip","chip":"esp32"}
a coroutine was expected, got <bound method WsOtaHandler.start_ of <WsOtaHandler(thread_ota, started daemon 140227056555776)>>
Traceback (most recent call last):
  File "server.py", line 93, in run
    asyncio.run(self.start_)
  File "/usr/lib/python3.8/asyncio/runners.py", line 37, in run
    raise ValueError("a coroutine was expected, got {!r}".format(main))
ValueError: a coroutine was expected, got <bound method WsOtaHandler.start_ of <WsOtaHandler(thread_ota, started daemon 140227056555776)>>

any idea whats wrong? thats error once esp32 connects to WS server

Tony763 commented 2 years ago

Hi @dr3adx, I made a small mistake on line 97. Change asyncio.run(self.start_) to asyncio.run(self.start_())

I will send a PR with the fix.

Tony763 commented 2 years ago

Hi @dr3adx, did change fixed run for You?

Hokyjack commented 2 years ago

Sharing my approach to WebSocket OTA update, the WebSocket client is just a small wrapper around the websocket class, the Operation is just an enumeration for int. Might help someone.

WebsocketClient * client;
int otaPartsLeft = 0;
bool otaUpdatingInProgress = false;

void webSocketEvent(WStype_t type, uint8_t *payload, size_t length) {
    switch (type) {

        case WStype_CONNECTED:

            if (otaUpdatingInProgress) {
                Serial.println("ERROR: when OTA updating, should restart device");
                otaUpdatingInProgress = false;
            }
            break;

        case WStype_TEXT: {
            Serial.printf("[WSc] Recieved text: %s\n", payload);
            int operation = client->getOperation();
            // Serial.printf("Operation code: %d\n", operation);

            // multi-message operations
            if (operation != NO_OP) {

                switch (operation) {
                    case OTA_UPDATE: {
                        String msg = String((const char *)payload);
                        if (msg == "done") {
                            // client->closeFile();
                            Serial.println("Firmware file closed!");

                            client->setOperation(NO_OP);
                            Serial.println("All parts written, running Update.end()");

                            if (Update.end()) {
                                Serial.println("Update end OK, restarting ...");
                                otaUpdatingInProgress = false;
                                delay(1000);
                                ESP.restart();
                            } else {
                                Serial.println("Update not OK");
                                Update.printError(Serial);
                                otaUpdatingInProgress = false;
                                client->setOperation(NO_OP);
                            }
                            break;
                        }
                        if (!otaUpdatingInProgress) {

                            // expecting message in format <LONG SIZE>:<MD5 CHECKSUM>
                            otaUpdatingInProgress = true;

                            String otaSizeStr = getValue(msg, ':', 0);
                            String otaMd5 = getValue(msg, ':', 1);
                            unsigned long otaSize = atol(otaSizeStr.c_str());

                            Serial.print("otaSize: "); Serial.println(otaSize);
                            Serial.print("otaMd5: "); Serial.println(otaMd5);
                            // if (!client->isFileOpened()) {

                            if (!Update.begin(otaSize, 0)) {
                                Serial.println("Update Begin Error");
                                Update.printError(Serial);
                                otaUpdatingInProgress = false;
                                client->setOperation(NO_OP);
                                return;
                            }
                            Update.setMD5(otaMd5.c_str());

                            Serial.println("Update Begin OK");
                            client->sendOtaChunkReady();
                        } else {
                            Serial.println("OTA Updating already in progress. Skipping...");
                        }

                    }
                        break;

                    default:break;
                }
                // if operation is none - get operation code from string (onetime operations)
            } 
        }
            break;

        case WStype_BIN: {
            Serial.printf("[WSc] Recieved binary length: %u\n", length)

                case OTA_UPDATE : {
                     // client->writeDataToFile(payload, length);
                        Serial.println("OTA recieving " + String(length) + " bytes");
                        Update.write(payload, length);
                        if (Update.isFinished()) {
                            Serial.println("Update finished");
                        } else {
                            client->sendOtaChunkReady();
                        }

                } break;

                default:break;
            }
        }
    }
}

void setup() {
    client = new WebsocketClient(webSocketEvent);
    client->setOperation(NO_OP);
}

void loop() {
    // Handle Websocket events if WiFi online
    if (client->isConnected()) {
        client->loop();
    } else {
        client->disconnect();
    }
}
emanavas commented 1 year ago

Sharing my approach to WebSocket OTA update, the WebSocket client is just a small wrapper around the websocket class, the Operation is just an enumeration for int. Might help someone.

WebsocketClient * client;
int otaPartsLeft = 0;
bool otaUpdatingInProgress = false;

void webSocketEvent(WStype_t type, uint8_t *payload, size_t length) {
    switch (type) {

        case WStype_CONNECTED:

            if (otaUpdatingInProgress) {
                Serial.println("ERROR: when OTA updating, should restart device");
                otaUpdatingInProgress = false;
            }
            break;

        case WStype_TEXT: {
            Serial.printf("[WSc] Recieved text: %s\n", payload);
            int operation = client->getOperation();
            // Serial.printf("Operation code: %d\n", operation);

            // multi-message operations
            if (operation != NO_OP) {

                switch (operation) {
                    case OTA_UPDATE: {
                        String msg = String((const char *)payload);
                        if (msg == "done") {
                            // client->closeFile();
                            Serial.println("Firmware file closed!");

                            client->setOperation(NO_OP);
                            Serial.println("All parts written, running Update.end()");

                            if (Update.end()) {
                                Serial.println("Update end OK, restarting ...");
                                otaUpdatingInProgress = false;
                                delay(1000);
                                ESP.restart();
                            } else {
                                Serial.println("Update not OK");
                                Update.printError(Serial);
                                otaUpdatingInProgress = false;
                                client->setOperation(NO_OP);
                            }
                            break;
                        }
                        if (!otaUpdatingInProgress) {

                            // expecting message in format <LONG SIZE>:<MD5 CHECKSUM>
                            otaUpdatingInProgress = true;

                            String otaSizeStr = getValue(msg, ':', 0);
                            String otaMd5 = getValue(msg, ':', 1);
                            unsigned long otaSize = atol(otaSizeStr.c_str());

                            Serial.print("otaSize: "); Serial.println(otaSize);
                            Serial.print("otaMd5: "); Serial.println(otaMd5);
                            // if (!client->isFileOpened()) {

                            if (!Update.begin(otaSize, 0)) {
                                Serial.println("Update Begin Error");
                                Update.printError(Serial);
                                otaUpdatingInProgress = false;
                                client->setOperation(NO_OP);
                                return;
                            }
                            Update.setMD5(otaMd5.c_str());

                            Serial.println("Update Begin OK");
                            client->sendOtaChunkReady();
                        } else {
                            Serial.println("OTA Updating already in progress. Skipping...");
                        }

                    }
                        break;

                    default:break;
                }
                // if operation is none - get operation code from string (onetime operations)
            } 
        }
            break;

        case WStype_BIN: {
            Serial.printf("[WSc] Recieved binary length: %u\n", length)

                case OTA_UPDATE : {
                     // client->writeDataToFile(payload, length);
                        Serial.println("OTA recieving " + String(length) + " bytes");
                        Update.write(payload, length);
                        if (Update.isFinished()) {
                            Serial.println("Update finished");
                        } else {
                            client->sendOtaChunkReady();
                        }

                } break;

                default:break;
            }
        }
    }
}

void setup() {
    client = new WebsocketClient(webSocketEvent);
    client->setOperation(NO_OP);
}

void loop() {
    // Handle Websocket events if WiFi online
    if (client->isConnected()) {
        client->loop();
    } else {
        client->disconnect();
    }
}

Hi @Hokyjack , I triying to do OTA winth WS too, but I cant find the function "setOperation(NO_OP)" and "getOperation()". Do you get any positive result with this code?