esp_delta_ota fails with error "esp_delta_ota: Error while applying patch: Function not implemented." (IEC-113)

[ntremble-sst]

Answers checklist.

• [X] I have read the documentation of the component in question and the issue is not addressed there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi,

I created the esp_delta_ota example project and had the HTTP sample work correctly on the ESP32-S3 target.

I then attempted to incorporate that code into my project using the same IDF 5.1.2 as I used for the example, but it does not complete. The primary differences are:

• the binaries are much larger (5MB). My test has only a small change, but the patch comes out at 89KB heatshrink compressed.
• the patch is being read from an eMMC card, having been received via CAN at some earlier point. Standard POSIX file i/o.

When I run the code it commences feeding the first byte after the 64 byte header (magic+SHA256+reserved). It gets in about a dozen bytes, then the detools function dfpatch_size() returns 'not implemented' with a size value of 656.

I have tried different sized blocks (64 bytes, 1024 bytes, 2048 bytes) but always the same error.

This log shows what is happening. I have added some logging to detools, so the line numbers will be a little out compared to the pure code, but only by a few lines. I can fix that if it helps.

I found it necessary to modify the python script somewhat to get it to run on Windows, but I did try running the original on Linux and got the same output, so pretty sure that's not the problem.

esp_delta_ota 1.10 direct from the component library as a managed component. detools 0.53 (part of the same component) heatshrink 0.4.1 (part of the same component) esp-idf 5.1.2 python 3.11.2 ESP32-S3-WROOM-1-N16R8

Thank you.

d (67496) OTA: enUpdateFirmware(): feeding block to patch updater detools process_init() line 1221: entry detools chunk_get_no_check() line 87: data: 0x04 detools chunk_get_no_check() line 87: data: 0x80 detools chunk_get_no_check() line 87: data: 0xA7 detools chunk_get_no_check() line 87: data: 0xAC detools chunk_get_no_check() line 87: data: 0x05 detools process_dfpatch_size() line 1249: entry detools patch_reader_decompress() line 1051: call algorithm specific decompress() detools chunk_get_no_check() line 87: data: 0x44 detools chunk_get_no_check() line 87: data: 0xC8 detools chunk_get_no_check() line 87: data: 0x42 detools patch_reader_decompress() line 1051: call algorithm specific decompress() detools chunk_get_no_check() line 87: data: 0xA0 detools process_dfpatch_size() line 1259: size 656 not implemented E (67549) esp_delta_ota: Error while applying patch: Function not implemented.

[ntremble-sst]

The following files are attached as a single zip file:

• a log from the patch generation, including detools info output.
• a putty log of a failed patch application. The actual patch application commences at line 91, failure occurs at line 283.
• my modified detools.c/h, redone such that line numbers match the original unmodified file.

I've also included the patch file as a separate zip in case it is of use.

Note I have set up a Kconfig to control the definitions that were in the detools CMakeLists.txt (now commented out), you will see the changes related to that at the top of detools.h. They are working because (a) I can see the included/excluded code regions in my IDE, and (b) I did set breakpoints to confirm the correct [heatshrink] code was being reached.

The putty log includes a couple of dumps, being the header (magic+SHA256+reserved=64 bytes) and the first block of data from the actual patch as generated by detools.

As a slight aside, how hard would it be to add support for better compression? My test show that e.g. LZMA compression performs far better, yielding patch files in the hundreds of bytes range, but unfortunately, while provided for by the code, it is not actually implemented. For our application the size of heatshrink compressed patches is likely to be a real issue as our updates will be transferred over a cellular link. We have plenty of storage, but bigger files have more chance of being interrupted during transfer, as well as the data cost considerations.

my-files.zip patch-95-96.zip

[ntremble-sst]

I just tried feeding our patch file through the Espressif example code. Same result. See attached image.

[ntremble-sst]

I found an issue in my code, and now it appears to be working. I will close this issue and reopen/recreate if necessary.

[ntremble-sst]

I have determined the issue is in the patch generation i.e. the python script.

I would expect that given the same two input files the patch generation script should produce identical output on every run, so I should be able to take the sample old and new binaries and produce the identical patch file.

Using the sample binary and patch files provided with the https_delta_ota example (part of the esp_delta_ota v1.10 component) I am unable to regenerate the patch file using a fresh ESP-IDF v5.1.2 installation on a Debian 12.5.0 virtual machine (or on Windows 10).

The screen shot below shows my problem. If I use the sample binaries and attempt to recreate the identical patch file I get a python error, and the patch generation fails. That says to me that the python script as supplied is not the one that was used to create that sample patch file - or perhaps it was, but the environment has changed such that it no longer works.

I hit this same issue on Windows, where I modified the script to make it work, but it still does not generate the same patch file given the same input files. Note I specified --chip esp32s3 as that is what I require, but I did also try with --chip esp32, that was no better.

This leads to my earlier issue, as the patch I am generating is in some manner faulty, causing the patch process to fail.

[hmalpani]

Hello @ntremble-sst The python script esp_delta_ota_patch_gen.py is used to create patch file for ESP-IDF application binaries. It parses some data in the old binary which is used in header of the patch file created. Therefore that python script cannot be used for other files. I recommend you directly use detools command as follows: detools create_patch -c heatshrink <base_binary> <new_binary> <patch_file_name>

I think we can update the documentation so that such confusion doesn't occur in future.

[ntremble-sst]

Hello @hmalpani,

Yes, I spent quite some time working out what that script does, and I am using a cut down version of it to generate the header (magic+SHA256+28 bytes reserved) for my own binaries. There does seem to be an issue with the way the python script calls detools however, so I wound up using python to create the header and a direct call to detools to create the patch, followed by a concatenation step. I am now successfully generating patch files for my own binaries with the same structure, but obviously with different hashes in the header.

It would be good if the documentation was updated a bit.

What would be really good would be if the other compression methods were supported by the IDF for decompression. For our large binaries heatshrink does not do a particularly good job. My tests indicate LZMA would be much better. Unfortunately, although the conditionals are there, some code is missing.

Thank you for your response.

[ntremble-sst]

Hello @hmalpani,

Just to let you know I have it working now for any pair of binary files I care to feed it.

I will close this issue.