support for firmware update through home assistant

IgorYbema commented 2 months ago

As the developer of the Heishamon software I would like to discuss the posibility to add upgrading through the home assistant heishamon integration. I am not a python developer and am not a frequent user of home assistant. But I like where it is going and there a lot of users using it.

So what could I do on the heishamon side to support firmware upgrades through this integration?

Currently heishamon firmware updates is done using a POST to the /firmware url with filename and md5 as the post values, including the firmware itself ofcourse. Is this something which the integration could post a self-downloaded (from github) file to?

I can't make OTA upgrade on heishamon as the ESP8266 is too low on memory to add HTTPS support (required for it to download the binaries from github). I could host the binaries on a HTTP site but that would make it depended on that, probably, single server. That is why I wanted to look into the home assistant method of upgrading the firmware.

geduxas commented 2 months ago

HomeAssistant has some powerful tools like Esphome, its portainer based ESP compiler and local OTA update tool.. i think it's possible to use it..

geduxas commented 2 months ago

Also, HA is mostly based on docker, and have containers as add-ons, its possible to make some add-on's which task would be update heishamon

geduxas commented 2 months ago

And definitely i think it's possible download and simple post into heishamon interface :D

kamaradclimber commented 2 months ago

Hey there, it seems like a really good feature idea.

Two things makes me hesitate though:

my experience with upgrading heishamon has not been great so far (I tried to upgrade 2 days ago and the upgrade failed silently without the new version been applied) => the upgrade process would need to send feedback for the integration
once the integration does the upgrade, it means users will turn to this repo for support when upgrades does not work or brick their device. => heishamon needs to send info to select perfectly the type of binary to download (small or large).

But overall, I would really happy to work in that direction, thanks for the proposal!

IgorYbema commented 2 months ago

Good points. Let me think about those things.

IgorYbema commented 2 months ago

I have been testing and a basic python code upload that works below. This already results in a final answer succesful , rebooting or error MD5 wrong or like that.

import requests

# Read MD5 hash from a .md5 file
def read_md5(md5_file_path):
    with open(md5_file_path, "r") as f:
        md5_hash = f.read().strip()  # Read the MD5 and strip any extra spaces or newlines
    return md5_hash

# Paths for the firmware file and the .md5 file
file_path = "HeishaMon.ino.d1-v3.8.bin"
md5_file_path = "HeishaMon.ino.d1-v3.8.md5"
url = "http://heishamontest.local/firmware"

# Read MD5 hash from the .md5 file
md5_hash = read_md5(md5_file_path)

# Open the firmware file in binary mode
with open(file_path, 'rb') as firmware_file:
    # Prepare the multipart form-data
    files = {
        'firmware': ('firmware.bin',firmware_file,'application/octet-stream')
    }
    data = {
        'md5': md5_hash
    }

    # Send the POST request
    response = requests.post(url, files=files, data=data)

# Check the response from the server
if response.status_code == 200:
    print("File uploaded successfully!")
    print(response.text)
else:
    print(f"Failed to upload file. Status code: {response.status_code}")
    print(response.text)

kamaradclimber commented 2 months ago

Thanks for the script, it's a very good starting point.

I retried to update my firmware (from 3.0 to 3.8) but aftr the upload it fails to do anything (the script times out). On the console:

Tue Sep 10 16:03:56 2024 (274781014): Uploading new firmware: 95%
Tue Sep 10 16:03:57 2024 (274781148): Heishamon stats: Uptime: 3 days 4 hours 19 minutes 41 seconds ## Free memory: 58% ## Heap fragmentation: 10% ## Max free block: 21080 bytes ## Free heap: 23408 bytes ## Wifi: 100% (RSSI: -50) ## Mqtt reconnects: 2888 ## Correct data: 99.99%
Tue Sep 10 16:03:57 2024 (274781152): Requesting new panasonic data
Tue Sep 10 16:03:57 2024 (274781153): sent bytes: 111 including checksum value: 18 
Tue Sep 10 16:03:57 2024 (274781209): Received 203 bytes data
Tue Sep 10 16:03:57 2024 (274781210): Checksum and header received ok!
Tue Sep 10 16:03:57 2024 (274781216): received TOP33 Room_Thermostat_Temp: 26
Tue Sep 10 16:03:59 2024 (274783226): Readlen: 672973, Totallen: 674628
Tue Sep 10 16:03:59 2024 (274783236): Readlen: 674639, Totallen: 674628
Tue Sep 10 16:03:59 2024 (274783241): Uploading new firmware: 100%
Tue Sep 10 16:03:59 2024 (274783242): Readlen: 674736, Totallen: 674628
Tue Sep 10 16:03:59 2024 (274783251): Readlen: 674776, Totallen: 674628

(goal of this ticket is not to troubleshoot my issue but I'll need to get this fixed to be able to develop the feature in the integration). I'll check how to move to 3.6 with a serial cable (never done it but it should doable) where I think the bug has been fixed (I assume https://github.com/Egyras/HeishaMon/commit/334c165a8f48b72e8ad0bf53b4eaa5a9736e521a is the solution)

IgorYbema commented 2 months ago

There are some issue's with older firmware while uploading the firmware indeed. But most of the time, if you reboot and then do the firmware update directly from the webinterface it should work. I haven't tried 3.0 to 3.8 yet myself though.

You could easily use https://espressif.github.io/esptool-js/ to upload the binary over serial (make sure to upload to address 0x0) on the ESP8266. For the ESP32 it is a bit more work.

I think we should support updating from home assistant only from a firmware where we believe it is working correctly. Also indeed it should tell what controller it is running (and so, which firmware it needs). So that should be in the next firmware first (I am thinking about a GET /firmware?status request giving some info in JSON format.

IgorYbema commented 2 months ago

Just tried a v3.0 to v3.8 and it worked fine, right after reboot:

Tue Sep 10 16:38:42 2024 (73388): Uploading new firmware: 90%
Tue Sep 10 16:38:42 2024 (74074): Uploading new firmware: 95%
Tue Sep 10 16:38:43 2024 (74782): Readlen: 672640, Totallen: 674627
Tue Sep 10 16:38:43 2024 (74790): Readlen: 673116, Totallen: 674627
Tue Sep 10 16:38:43 2024 (74799): Readlen: 674627, Totallen: 674627
Tue Sep 10 16:38:43 2024 (74802): In /firmware client write part
Tue Sep 10 16:38:43 2024 (74854): Firmware update success
Tue Sep 10 16:38:43 2024 (74863): In /firmware client write part
Tue Sep 10 16:38:43 2024 (74867): In /firmware client write part

kamaradclimber commented 2 months ago

I flashed to 3.8 with a cable. Then any attempt to use the script (to downgrade for instance to 3.6) ends up with

Remote end closed connection without response

(roughly at the moment we reached 100%).

I've tried to do the upgrade attempt shortly after reboot but no luck so far.

geduxas commented 2 months ago

@kamaradclimber do you have correct md5 file? Also does it fit current firmware? If you try to flash firmware without correct md5 it just reboots after upload. Don't remember it will pronounce any error or not...

kamaradclimber commented 2 months ago

Yes md5 files are correct and fit the firmware (both for 3.8 and 3.62). I've tried to remove sending the md5 from the script, no change.

To be clear, the script is not the issue, the firmware upgrade does not work better from the webui.

Should I open a ticket on Egyras/Heishamon repository to summarize the issue so that we can investigate independantly of this feature? (I think I can implement a draft without having a way to test)

geduxas commented 2 months ago

I'll test @IgorYbema script also to be shure.. as he told it's simple http post process. Maybe we are missing something.. also if there is any kind of problem from heishamon side @IgorYbema is only one who could solve it ;)

geduxas commented 2 months ago

yes indeed, @IgorYbema there is a problem with web stack. If i open console web interface i am unable to update firmware. Heishamon will interrupt upload, and closes socket, also i see MQTT reconnect. Next time while trying it will get ~5% of data and does same. After closing web browser and uploading, everything works good.

Console was opened from one device, and python script was run from another. if it makes any sense..

UPD: it doesn't reconnects to MQTT, just after closing upload process it just updates MQTT topic.

IgorYbema commented 2 months ago

Ok I'll try that also. Will be tomorrow.

IgorYbema commented 2 months ago

I saw same behaviour indeed. Something to fix :)

kamaradclimber commented 2 months ago

It describes exactly what I was facing (looking at the console to follow the update). Thanks for the confirmation. I’ll retry to update without looking at the console and start the implementation this weekend probably.

IgorYbema commented 2 months ago

I believe the websocket implementation is corrupting the memory somewhere. So it mangels some bytes being sent over another webserver connection. I'll take this finding back to my repo and discuss it with Curlymoo.

kamaradclimber commented 2 months ago

heishamon needs to send info to select perfectly the type of binary to download (small or large).

Is there anyway to distinguish the small and large versions of the PCB (with current version of heishamon), something like a mqtt topic that would exist only for one of those models?

In future version of heishamon it would be good to give the info about the type of build the board is running.

kamaradclimber commented 2 months ago

I've written https://github.com/kamaradclimber/heishamon-homeassistant/pull/249 as a draft.

For now it assumes the model is "model-type-small"
It "works" up to the point the firmware is uploaded (and then heishamon brutally cuts the connection and does not reboot, I don't know how to troubleshoot this).

Any way it's a good start.

If you want to test it easily, you can use the /developer-tools/action page on HA that allows to use the heishamon_firmware.update action and specify any version.

kamaradclimber commented 1 week ago

Would you be able to look at https://github.com/IgorYbema/HeishaMon/pull/140 and tell me if that would be a good idea for Heishamon ?

kamaradclimber / heishamon-homeassistant

support for firmware update through home assistant #248