OTA update via HTTP

[guestisp]

Is be possible to implement automated (or semi-automated, triggered via api/mqtt command) off-site OTA updates via HTTP ?

Something similiar to this sample sketch: https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/examples/AWS_S3_OTA_Update/AWS_S3_OTA_Update.ino

where a new firmware is hosted on a remote server and fetched by esphome for upgrades.

Even better would be using automated updates, like fetching a remote url with some arguments (ie https://my.remote.server/update?v=1.2.3&h=esphomenodename&key=verysecretkey) and then the remote server returns the bin file if update is available or 404 if nothing to do. This call could be placed in a loop on esphome

[OttoWinter]

What would be the use case for this?

You need to compile the binary for each device anyway, and the OTA "pushing" can easily be automated with a quick bash script.

[guestisp]

Use case are remote unattended upgrades without direct access to device. A firmware update is published on remote server and each nodes could update automatically

[OttoWinter]

@guestisp Yeah but the existing OTA mechanism already does most of that, no?

With one command the update is uploaded to the ESP. Then you can also create a quick bash script that performs an OTA on all devices in your network.

[guestisp]

Thats the point: in my network Im talking about downloading a firmware upgrade from an external server allowing unattended upgrade without the need of a PC on the same network

A customer calls, i publish and update on my server and automatically the customer esphome start the download

[ajcollett]

Maybe I can jump in, I guess the issue is where the upgrade is initiated from, because you might not be able to reach the remote device behind a firewall or not know it's IP, but it can reach an external server and pull the update?

[guestisp]

Exactly

[skorokithakis]

I agree with this, I use this with ESPurna to update my parents' house devices, and it works great.

I use espota-server, which is a simple generic server, you drop the a file in the directory called firmwarename-version.bin, the device checks for new firmware every so often and if a newer one is found, it automatically updates.

This way I don't even need to know the IP addresses of the devices, they'll just all autoupdate whenever I compile a new firmware version.

[jperquin]

I have a similar use case. Devices run MQTT offsite behind a firewall where they are not (easily) reachable. Every so often I'd like to have device specific (not just generic) fw updates executed..

[jperquin]

@skorokithakis looks like you already cracked this nut..

Could you share an example of your .yaml for reference?

Thanks in advance.

[skorokithakis]

Sorry, I meant it's Espurna that autoupdates (ESPhome doesn't support this yet AFAICT), so I have no YAML to share.

[jperquin]

ESPHOME is such a versatile platform that has great potential outside the home (ie. your own network), especially when using MQTT for connecting in from remote sites (behind their own firewall).

An important shortcoming is the fact that once an ESP device with ESPHOME is deployed to a remote site, OTA firmware updates are no longer possible unless you punch a hole in the remote firewall (not possible in most cases) or physically travel to the site.

I am very appreciative of all the hard work done here by volunteers, but if there is a way to commission someone to develop this functionality, I'd be happy to pitch in..

Any thoughts from this wonderful community would be very welcome.

[skorokithakis]

I agree, please prioritize this. It's currently impossible to upgrade remote devices because of this, and being able to drop a firmware file on an ESPOTA-server instance and have tens of devices automatically update would be great. The ESP framework has built-in support for this, AFAIK, so it probably wouldn't even be too hard to implement.

[wizzor]

There is another request similar to this one, asking for MQTT based unattended upgrade.

Either one will work for my case, commenting so I will be notified if it's built.

[fuzzybear62]

No news or plans on this feature?

[chatziko]

A tutorial for implementing this feature in esp32, in case it's helpful: https://github.com/kurimawxx00/webota-esp32

Super useful feature, shouldn't be too hard to implement in esphome.

[smartynov]

I'm also looking for this feature. I'm going to place some esp8266-based sensors on a wifi network that "isolates" devices (an office space in my case). I cannot place the server there, nor can I control the port forwarding or any other network settings. Thus I set up a remote server, and while remote mqtt works ok, I still need to be able to OTA-upgrade the firmware.

[javawizard]

Same. What would be especially cool is if there was a way for the ESPHome Dashboard to push the firmware up somewhere public (configurable S3 bucket perhaps, or serve it in a way that nodes can download it through a user's external Home Assistant URL if they're using ESPHome with HA and have their instance publicly exposed through HA Cloud or otherwise) and then tell the node to update via MQTT - so the update process would remain the same one-click experience it is for nodes that are on the same network.

This might actually be a fun first contribution to ESPHome if I get the time...

[drunkly]

Does someone implement a custom componetn to do this?. I'm trying to implement on my own side by now but having troubles with libaries dependencies:

includes:
    - custom/brownout_disable.h
    - custom/custom_htud1d.h
    - custom/httpupdate.h
  libraries:
    - "Wire" #FOR I2C library
    - "SparkFun HTU21D Humidity and Temperature Sensor Breakout" #To read this fucking sensor 
    - "Wifi" #For http update
    - "HTTPClient"
    - "Update"
    #- "WiFiClientSecure" # for http update?
    #- "ESP32httpUpdate" #http update
    #- "HttpClient" # for http update
    #- "ArduinoJson" #for http update
    #- "PageBuilder" #for http update
    #- "ESP32-PSRamFS" #for http update
    #- "AutoConnect"
#RUN arduino-cli lib install "HTTPClient"
#RUN arduino-cli lib install "WiFiMulti"
    #- "StreamUtils"

//#include <HTTPUpdate.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <Update.h>

class HttpUpdate : public Component {
    private:
        static WiFiClientSecure wifi;
        static WiFiClient client;;

    public:
        uint32_t brown_reg_temp;
        float get_setup_priority() const override { return esphome::setup_priority::LATE; }

        int updateFirmware(String server, String port, String path_to_file){

            //ESPhttpUpdate.followRedirects(true) 
            //HttpUpdate::client.setFollowRedirects(followRedirects_t::HTTPC_STRICT_FOLLOW_REDIRECTS);
            //httpUpdate.setFollowRedirects(followRedirects_t::HTTPC_STRICT_FOLLOW_REDIRECTS);
            //Api::wifi.setCACert(ca_cert);

            t_httpUpdate_return ret = httpUpdate.update( HttpUpdate::wifi, server, port.as<uint16_t>(), path_to_file, "0");
            //t_httpUpdate_return ret = httpUpdate.update(client, "server", 80, "/file.bin");

            switch (ret) {
                case HTTP_UPDATE_FAILED:
                    //Serial.printf("HTTP_UPDATE_FAILED Error (%d): %s\n", httpUpdate.getLastError(), httpUpdate.getLastErrorString().c_str());
                    break;

                case HTTP_UPDATE_NO_UPDATES:
                    //Serial.println("HTTP_UPDATE_NO_UPDATES");
                    break;

                case HTTP_UPDATE_OK:
                    //Serial.println("HTTP_UPDATE_OK");
                break;
            }
            return ret;
        }

};

HttpUpdate * htttpupdate = new HttpUpdate();

.piolibdeps/limonero/ESP32-PSRamFS/src/PSRamFS.h:30:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pioenvs\limonero\libfc0\ESP32-PSRamFS\PSRamFS.cpp.o] Error 1
In file included from src/httpupdate.h:6,
                 from src/main.cpp:323:
.piolibdeps/limonero/ESP32httpUpdate/src/ESP32HTTPUpdate.h:36:10: fatal error: FS.h: No such file or directory

************************************************************
* Looking for FS.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:FS.h"
* Web  > https://registry.platformio.org/search?q=header:FS.h
*
************************************************************

 #include "FS.h"
          ^~~~~~
compilation terminated.
*** [.pioenvs\limonero\src\main.cpp.o] Error 1
In file included from .piolibdeps/limonero/ESP32httpUpdate/src/ESP32httpUpdate.cpp:26:
.piolibdeps/limonero/ESP32httpUpdate/src/ESP32httpUpdate.h:33:10: fatal error: HTTPClient.h: No such file or directory

********************************************************************
* Looking for HTTPClient.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:HTTPClient.h"
* Web  > https://registry.platformio.org/search?q=header:HTTPClient.h
*
********************************************************************

 #include <HTTPClient.h>
          ^~~~~~~~~~~~~~
compilation terminated.
*** [.pioenvs\limonero\lib246\ESP32httpUpdate\ESP32httpUpdate.cpp.o] Error 1
In file included from .piolibdeps/limonero/PageBuilder/src/PageBuilder.cpp:12:
.piolibdeps/limonero/PageBuilder/src/PageBuilder.h:26:10: fatal error: WebServer.h: No such file or directory

*******************************************************************
* Looking for WebServer.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:WebServer.h"
* Web  > https://registry.platformio.org/search?q=header:WebServer.h
*
*******************************************************************

 #include <WebServer.h>
          ^~~~~~~~~~~~~
compilation terminated.
*** [.pioenvs\limonero\lib16a\PageBuilder\PageBuilder.cpp.o] Error 1
============================================= [FAILED] Took 98.70 seconds =============================================```

[leoddias]

@oarcher already created a external component to do this. (I will test it later) https://github.com/oarcher/piotech/tree/main/components/ota_http

Just sharing and watching here for next update on this, it would be amazing to have this feature soon.

[Alphaemef]

This is really something I would love as well. How do we support an issue this old :) ?

@leoddias, the solution by @oarcher is really awesome, but the problem is it doesn't support ESP-IDF, which means we can't have SSL. Would be really awesome if there was a port.

[oarcher]

@Alphaemef , I've implemented ESP-IDF, and it works for SSL. Note that for the moment, RP2040 support is missing, but I plan to work on it soon.

[nagyrobi]

@oarcher is the url: http://example.com/firmware.bin templatable? Eg. something like url: !lambda return id(text_sensor).state; so we can dynamically adjust the URL if we want?

[oarcher]

@nagyrobi , yes, the url is templatable.

[nagyrobi]

Looking for some docs...

[oarcher]

I've updated the doc on https://github.com/esphome/esphome-docs/pull/3291 .

[Alphaemef]

@Alphaemef , I've implemented ESP-IDF, and it works for SSL. Note that for the moment, RP2040 support is missing, but I plan to work on it soon.

dude this is tremendous thank you so much for the effort!!! Its a game changer for me. Thanks!

[thomasvnl]

@oarcher this is really cool, thanks for implementing this. Is there any chance it will support functionality for version checking / hash comparison in the (near) future? Instead of a button, then you could have updates triggered by a timer and only install/update if there actually is an update. Otherwise, the "remote" aspect is still limited by a remote entity having to push a button (as per your documentation).

[oarcher]

@thomasvnl , I think that if you put the firmware md5sum on the server (like http://exemple.com/firmware.md5), you can check for change with http_request and trigger ota_http.flash action.

You will have to save the md5 in a global variable with restore_value: true (see https://esphome.io/guides/automations.html#global-variables).

So I think there is no need to implement this in ota_http, but on the other side, I think I will put an option in ota_http for url_md5: http://exemple.com/firmware.md5, to be able to check that the md5 of the received data match the remote one. So perhaps I will also add var_md5: firmware_md5, where firwmare_md5 is the name of the global variable to store the md5 of the current firmware. This will help for those who want to check against remote md5.

[skorokithakis]

If you're going to do that, you may want to support the existing protocols (Arduino/NoFUSS) directly, so users can just run espota-server and have it work out of the box:

https://gitlab.com/stavros/espota-server

HTTP-based update has been implemented by various projects, it might be better to use one of those protocols rather than coming up with a new one.

[oarcher]

@skorokithakis , yes, if there is an existing protocol, ota_http should support it. But is it restricted to esp8266 ? I can see on https://registry.platformio.org/libraries/xose/NoFUSS that the device should send http headers whith names like X-ESP8266-*

header	description	example
X-ESP8266-MAC	Device MAC address	5C:CF:7F:8B:6B:26
X-ESP8266-DEVICE	Device type	SENSOR
X-ESP8266-VERSION	Application version	0.1.0
X-ESP8266-BUILD	Application build	611cdf3

[skorokithakis]

No, it just checks a hash and downloads a file, really. They just named it like that because the ESP32 didn't exist back then.

[nagyrobi]

These headers could be customizable for ESPHome...

[skorokithakis]

Sure, but then ESPHome wouldn't be compatible with the other servers. I'd be happy to customize them, though.

[Alphaemef]

@oarcher Hi again, I have a small pause at the fact that the *.bin files are basically clear text config files, to have lying around open web servers.

Do you have any ideas to work around that ? is it at all possible to assign the ESP32 unit a password to provide to the website its tries to download from ? Any other solution ?

[oarcher]

Basic auth works out of the box for arduino if you use url like https://user:password@example.com/firmware.bin.

For esp-idf, I've just pushed a commit on https://github.com/esphome/esphome/pull/5586 to allow the same behaviour.

I will also update the doc.

[Alphaemef]

Basic auth works out of the box for arduino if you use url like https://user:password@example.com/firmware.bin.

For esp-idf, I've just pushed a commit on esphome/esphome#5586 to allow the same behaviour.

I will also update the doc.

dude you are the closest thing to an actual rockstar for me. This will be my work-flow: greater folder with basic auth place *.bin in that folder the url: https://user:password@example.com/firmware.bin including auth will be transmitted through MQTT on a topic that only the individual user has access to with the same user / password as that user (no new knowledge is transmitted). start the OTA delete file + folder from webserver.

this is perfect, and makes sure that a user that gets compromised and gives MQTT access, don't also compromise the sourcefile of all other units that are updated after that. Perfect perfect perfect :)

and again: thanks!!!!

[Alphaemef]

and just for clarity: did you implement the basic auth in the current available repository as well ? Or only in the PR for main ESPHome branch ?

[oarcher]

It depends on the origin of the request, and I merge to the other repo with some delay. So for features discussed here or in the PR the external component declaration should be:

external_components:
    # - source: github://oarcher/piotech
    - source: github://pr#5586
      components: [ ota_http ]

[digaus]

@oarcher Thanks for your work

Since you have it running for esp-idf could you have a look at the web_server ota feature? Or is it completely different? Currently OTA via web_server is not supported when using esp-idf :/

[oarcher]

@digaus , OTA for web_server is different than ota_http , but @angelnu who is working on https://github.com/esphome/esphome/pull/5535 has some ideas to refactor the ota base component so it will be easiest to implement OTA for web_server. He made a draft at https://github.com/angelnu/esphome-1/pull/1, but it will need more work, and I think we will try to merge our work into the current esphome branch before working on that.

[digaus]

Thanks for the detailed information. This looks promising and I am looking forward to it :)

[dentra]

web_servert_idf OTA requires support of multipart post requests first. If someone wants to implement this I left a note in my last PR to web_server_idf.