OTA for nodemcu

[romanchyla]

Missing feature

OTA - Over the air update of the firmware

Justification

OTA is super important for devices that work in the field. It is not as important for hobbyists, but they might also enjoy little bit of freedom. Even though there are various mechanisms for updating lua scripts, the firmware still remains the same.

There exist two PR #806 and #816 that implement OTA.

I'm proposing to review both, test them and report back to community the results. Hopefully, in the form of PR that could be incorporated.

Requirements

Because of the ssh verify reported not working, I'd like to concentrate on the most recent SDK (so no backward compatibility).

Questions

Here is what I'd like to answer in this Issue.

1. test each implementation against current master (or dev branch) for esp-01 (1Mb version) esp-12 (32m version; dev kit)
2. describe details how each solution works and explain consequences
3. make a PR with a justification of why/what seems like the best to incorporate

I'll generate detailed questions. PLEASE add whatever concerns you would like to have addressed into the comments. Hopefully, the authors of the respective OTA implementation will be kind enough to help me with some questions/problems.

Also, I'd very interested in having lua snippets that exercise critical functionality. For example, if you know that certain cryptographic functions might be affected, it would be helpful to suggest ways to test them.

[romanchyla]

Hello @jmattsson and @raburton, would you mind assisting in this review?

[raburton]

Sure. If you have any specific questions I'll try and answer them.

[romanchyla]

Thanks @raburton; I've just made a PR in your repo - rebased your code on top of master, it compiles but I didn't yet have a chance to flash it.

[devyte]

Hi @romanchyla

rebased your code on top of master

for nodemcu, all PRs are to be made to the dev branch (i.e.: new features are implemented in the dev branch, while master is meant to represent a stable snapshot). I suggest you rebase on dev instead of master, it will make things easier once the feature is ready.

Alternatively, you could work on master, but once the feature is ready you will have to port it to the dev branch first before you make the PR.

[devyte]

My own personal feelings about this, without actually having tested the solutions (I've been meaning to do that for a while now): Advantages of raburton's solution:

• rboot is a known bootloader
• does image checksum, so if firmware is corrupted, it won't boot, which is a good thing

Advantages of jmattsson's solution:

• has in-test feature to prevent bricking
• allows spiffs to be shared between firmware (not sure if this is allowed in raburton's solution
• has an api that may be a bit more advanced/flexible than raburton's solution

jmattsson says they are using a minimal bootloader. My takeaway from that is that it's an in-house one (I could be wrong). Actually, whether using rboot is better or not vs. jmattsson's minimal bootloader is something that could be investigated as well. Adding a checksum to that minimal bootloader is probably trivial.

If we could get the best of both worlds, it would be awesome, but I would settle for just getting any OTA update functionality formally into the nodemcu firmware.

[raburton]

Not sure how the in-test mode works, but rBoot has a temporary boot mode which can be used to ensure you don't switch to a ROM that looks good but doesn't actually work in some functional way. This was added to rBoot after the integration with nodemcu, but should be easy enough to enable it. You can share spiffs or use separate ones, that's nothing to do with the bootloader and entirely a question for the application. I haven't compared the APIs, but what you say is probably true, I only added a trivial lua api for rBoot as a demonstration, ideally it would be improved or replaced by someone more familiar with the internals of nodemcu and what users actually want to do with it.

[romanchyla]

Thanks @devyte - I think I'll use master for now - I want to be sure that errors, if any, come from the OTA parts; I can make a PR against the dev branch

Good points, I'll include them in the exploration. @raburton explained the two (thank you!), I'll try to document the api differences

[jmattsson]

I haven't chimed in here for a while, but currently I'm waiting to see (and play with) the final details of the ESP32 loader/partitioning. There's already OTA support in that loader, but I'm not convinced their partitioning approach is the best when it comes to the OTA features. I'd consider needing a decicated block just to point out which image to use to be more error-prone than having status-bits built into the OTA image itself (the latter being the approach taken in "my" loader).

Once I understand more of the ESP32, I'll see what I can come up with that would be suitable across both chips. If @raburton adds ESP32 support to rBoot that would certainly be something I'd look at. Since this is something I'd be doing for $work, there'd likely be a strong preference towards the minimal end of the scale, since we're needing to conserve battery like there's no tomorrow, and spending (m)any unnecessary cycles in a loader is frowned upon.

TL;DR: I totally haven't forgotten about getting OTA support into NodeMCU officially, but now waiting on more ESP32 details & hands-on.

[romanchyla]

Hi @jmattsson,

The ESP32 seems really nice and I saw all the work all of you are putting into making nodemcu ready for it, thank you!

I'll try my best to help to integrate existing OTA work into the current nodemcu, since I think many people will be using ESP8266 for many months to come (and RTOS branch will be bleeding-edge for some time). If the esp OTA loader then takes over, all will still be good. But maybe rBoot will also evolve to support the ESP32 and then it might be a question "which of them?". I'll try to document all steps so that the decision might be taken quickly.

If I understand your comments about unnecessary cycles, you would prefer to have a bootloader that can support both chips (and in extension, if rBoot or your original solution could do that - it would be a very important argument). Can you please or @raburton share your opinion on this subject, once you form it?

[romanchyla]

I've started by studying the important concepts, it is work in progress: https://github.com/romanchyla/nodemcu-firmware/blob/ota/docs/en/ota.md

However, interesting find for @raburton and @jmattsson: Espressif open sourced their bootloader for ESP32 and it has the interesting OTA bits too: https://github.com/espressif/esp-idf/tree/master/components/bootloader/src/main

[helxsz]

any more progress?

[jmattsson]

The ESP32 is an altogether different beast to the ESP8266. I'm happy for the APIs to be different, they are in many other areas already. As for which OTA approach to use for the 8266, I don't want to weigh in too heavily since I can be seen to have a conflict of interest. #806 is field-proven at my $work, but I can see others preferring the expanded boot functionality of rBoot.

[djphoenix]

@jmattsson of course prefer, but not only. I have working (and production ready!) branch with integrated rBoot, and it works perfect with sdk200 branch too. Unfortunately it have a lot of changes (for build-files too), so I will find a way (and time) to simplify it and prepare a PR. But SDK200 and LWIP is my priorities right now.

[devyte]

Now that #1435 (sdk200) is merged, what is the next step for this? We currently seem to have two field-proven solutions available for OTA updating. I think that at this point either one is good enough. Has one been chosen already?

[djphoenix]

@devyte now I very busy on my primary work, and so on LWIP branch. But rboot-PR is in my own todo-list.

[romanchyla]

For what is worth, I'll also get to this soon (a hardware project took all my time, but it's almost done)

On Tue, Dec 13, 2016 at 7:13 AM, Yury Popov notifications@github.com wrote:

@devyte https://github.com/devyte now I very busy on my primary work, and so on LWIP branch. But rboot-PR is in my own todo-list.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodemcu/nodemcu-firmware/issues/1496#issuecomment-266723560, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZIkhbPvHRrMqJNihUVA5XrE6Z8en2aks5rHowCgaJpZM4J4T7p .

[devyte]

@romanchyla it's worth a lot :) I actually have 2 projects with development on-hold waiting on this one feature. If I weren't tied up elsewhere, I would have looked into it already.

[devyte]

Has there been any progress at all on this one, or not yet?

[djphoenix]

@devyte I have something to show in rboot branch in my fork, but work is incomplete at all. So there is nothing about self-update. Just two separate roms with ability to swap between them.

[joysfera]

It's been five months since last update so I thought I'd ping if there's any progress? dev-esp32 is becoming more and more usable so it'd be great if some kind of OTA update support was there sooner than later.

I myself would be happy with a manual firmware update approach where my webserver would receive a file, store it to my filesystem and then flash it to the other partition and at last would mark some flag somewhere to let bootloader start the other partition after next reboot. Simple and fully under control.

Then I would be interested in including lua scripts in the firmware update. I think it's possible to attach a SPIFFS to a nodemcu image but I haven't seen it documented.

[nwf]

@joysfera docs/en/spiffs.md documents the existing build infrastructure for pre-build SPIFFS images, FYI.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[poorandunlucky]

Any progress on bringing OTA capabilities to NodeMCU, what with the new node.setpartitiontable() functions and all..?

[TerryE]

Any progress on bringing OTA capabilities to NodeMCU

Still on my roadmap, but at the moment there's only one developer active on the core work (me) and my focus is getting Lua 53 with its on-ESP LFS building capability. So maybe by early 2020.

[raburton]

At least one complete OTA solution was provided by me nearly 4 years ago (#816) , and other people contributed other options as well. Seems there was insufficient interest (despite plenty of people in these threads) for any of them to get adopted.

[poorandunlucky]

Wow, that's great work 😟 A lot of people participated, too... you must've been proud, and it must have sucked hard it didn't get pulled, I can't see why it didn't... looks like it was ready, and everything was really nice and well documented and tested....

Jealous aspie just closed your PR? lol

---

From: Richard Antony Burton notifications@github.com Sent: Thursday, October 10, 2019, 02:47 To: nodemcu/nodemcu-firmware Cc: Patrick Dorion; Comment Subject: Re: [nodemcu/nodemcu-firmware] OTA for nodemcu (#1496)

At least one complete OTA solution was provided by me nearly 4 years ago (#816https://github.com/nodemcu/nodemcu-firmware/issues/816) , and other people contributed other options as well. Seems there was insufficient interest (despite plenty of people in these threads) for any of them to get adopted.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/nodemcu/nodemcu-firmware/issues/1496?email_source=notifications&email_token=AKMLVWKTSAKR7C5RMJQL5CLQN3FZTA5CNFSM4CPBH3U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA3ASMY#issuecomment-540412211, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKMLVWIHZCCRIRK7VFKRYD3QN3FZTANCNFSM4CPBH3UQ.

[TerryE]

@raburton Richard, we use OTA on our ESP32 branch and this is heavily influenced by your work, but our main challenge on the ESP8266 branch is what I could describe as the Sonoff requirement -- that is we are seeing the proliferation of Sonoff and Shelley devices that either use the ESP8266 + 8Mbit flash or the ESP8285 with its on SoC 8Mbit flash, so we really need an OTA solution that fits into a 1 Mb firmware. Your implementation doesn't help here, though the know-how and ideas are invaluable. Thanks.

[TerryE]

See #2606

[raburton]

Yes, anytime your flash is so small / image so large that you can't have two on there at the same time it's going to be a lot more difficult to do easy and resilient OTA. I wouldn't fancy trying to overwriting the running application. I'd go for an asymmetrical layout and have a small rom slot running a minimal non-lua rom that does the OTA and a larger slot for the actual lua rom.

[TerryE]

Yup, my thinking also. We'd need to gzip compress the image. If you look at our app/uzlib library, I've got a tiny inflator, and the SPIFFS read only library is about 8Kb, so my current thinking is to have a Lua module that wraps a ~16Kb boot loader which will run in the ICACHE region and read and inflate an .img.gz file over the top of the firmware. This is really a flash-to-flash operation, so the only real risk is powerfail during the ½sec or so needed to unpack the image. If this happens then it's back to the bootloader header.

Anyway, I am up to my eyes in the Lua 5.3 upgrade at the mo, but I hope to get this done before Xmas. If would be really valuable having a collaborator / expert reviewer who can tell me when my approach is 💩

[raburton]

Will the spiffs and uzlib libraries compile to run on baremetal or do they need to use functions from the esp sdk? Assuming you can get them to compile for baremetal (quite likely, but don't know how much work that would involve without a good look at the code) then you could include that functionality in to a simple bootloader and check/update the app on startup as required. That also assumes you can get enough compression on your rom image to fit it in a spiffs along side a decompressed rom slot. The principal seems sound. Happy to contribute if I can, but like you have many demands on my time!

[TerryE]

Will the spiffs and uzlib libraries compile to run on baremetal

Yes

or do they need to use functions from the esp sdk?

No, just the BOOT ROM API hooks.

you could include that functionality in to a simple bootloader and check/update the app on startup as required.

Yup, the firmware already has a bootstap startup with a boot-to-boot message passing service for configuration data. The current firmware implements a modified Harvard architecture for the Lua VM so that you can load up to 256Kb of Lua code and constant data into a flash region though 64Kb is enough for a lot of apps and this leaves the ~44Kb heap for data.

This bootloader would be somewhere between 16-24Kb so would fit comfortably in the 32Kb ICACHE IRAM0 segment if caching is disabled. I don't understand why you need a decompressed ROM slot as the inflator is a streaming implementation that would read directly from SPIFFS writing page-by-page overwriting the DRAM0, IRAM0 and IROM0 segments. Most of the know-how is available through /espressif/esptool/tree/master/flasher_stub.

The whole process is really just a smart flash-to-flash copy. We disable all interrupts and lock out the SDK during this window.

[raburton]

Will the spiffs and uzlib libraries compile to run on baremetal Yes

I've almost got uzlib compiling baremetal, but just trying to work out what the setjmp business is all about, and how to do without it as I don't think that's available.

I don't understand why you need a decompressed ROM slot as the inflator is a streaming implementation that would read directly from SPIFFS writing page-by-page overwriting the DRAM0, IRAM0 and IROM0 segments.

That's what I'm referring to when I say decompressed rom slot - the normal space taken up by the "installed" rom. If that's over half the size of the rom (hence not being able to do traditional two-rom OTA) and you need a spiffs containing a compressed version of the new rom (and user files), then you'll need a decent compression ratio.

The whole process is really just a smart flash-to-flash copy. We disable all interrupts and lock out the SDK during this window.

I'm confused about this. It makes it sound like you're replacing the lua rom while it's running, rather than from the bootloader, which is what I thought you were talking about. I thought you intended to download the compressed new rom to the spiffs, reboot and the bootloader would check the spiffs and install the new rom, then boot it (or on a normal boot, note the lack of ota image in the spiffs and just boot the existing rom).

[TerryE]

The main advantage of using SPIFFS is that we can use it as a churn FS storing app data, LFS images, or OTA images as needed.

It makes it sound like you're replacing the lua rom while it's running, rather than from the bootloader

Basically yes that is sort of an option. NodeMCU -- at least on the ESP8266 SDK build -- uses the ROM bootloader, and as we've done in the past we can slot in a user_start trampoline so that when we're on an OTA reboot we never actually enter the SDK or at least that's an option. The way that the SDK 3.0 starts up is different to 1.x and 2.x. It calls user_pre_init() with the icache enabled to allow the app to initialise the partition table, and the SDK itself doesn't start properly until control is returned. If doing the SPIFFS to firmware load, then we have the option of just taking over at this point and not handing back control to the SDK since we don't need any of the SDK services.

But on this path we would never actually start up the Lua VM, rather just drop into the inflate and copy code.

Note that this RO SPIFFS code is entirely disjoint from the full featured RW SPIFFS in the firmware. Most of the code is conditionally compiled out so this RO stub is only 6Kb or so (See SPIFFS binary sizing). It is also running with the icache disabled and can therefore be directly layered on top of the BootROM SPIflash API.

[raburton]

Gotcha, that makes sense then. It was the running code from the main rom before the sdk gets going that had me confused - not played with sdk 3.

[TerryE]

I will @ you when I post a PR. Thanks for your feedback. 👍

[jmattsson]

@raburton's right about the spiffs reading code needing to be integrated with the boot loader in this case though - it can't be part of the application that's about to get overwritten or everything will Really Not Work(tm). The only possible exception is if you hoist all of that code into IRAM and run the upgrade from there, in which case you could conceivably rewrite what you just booted, but that's quite fragile and can leave you with a non-bootable device if there's an interruption during the flash. If the boot loader is responsible for extracting the new image and getting into place, it can be written such that it will retry until it succeeds.

Something which I've done for some products in the past (though on Atmel chips) is to have the flash writing functions part of the boot loader (out of necessity), but then also expose them as library functions to the regular app. As in, link in the symbol addresses of the boot loader functions into the app so the app can directly call them. That's probably too fancy for what we're after, but could be considered.

[raburton]

Agreed @jmattsson, having thought a bit more about this overnight I'm back to thinking this must be fully separate (I was starting to think otherwise about user_pre_init()). Including this code in the user_pre_init() isn't the same as running baremetal. The SDK may not have started up all the fancy stuff, like wifi, but you're still using the SDK libraries. You can't easily get all the SDK code you call (and the code it calls that you don't know about) to run from IRAM. Even if you did you have the problem that if any error occurs, when you come to reboot you may have rewritten half of the image and you may have lost the portion of the image that includes user_pre_init() or something that it uses from SDK libraries, or they may now be in a different place. The code all needs to run from a separate app (a proper separate bootloader) from a separate bit of flash.

[TerryE]

No you aren't using the SDK libraries. Have a look at the Espressif flasher stub that I referenced. It is linked as a standalone stub using its own ld file and iram : org = 0x4010E000, len = 0x2000. The final binary might be included in an asm ( .incbin ) so that it is located as a contiguous block in RAM at a known location, but you bootstrap into this using a tiny IRAM1 hook which reads this block into RAM, turns off the cache, moves the stub into its correct base address in the now available IRAM0 and then executes it. Since the loader is now executing as a standalone stub in IRAM0, it can safely unpack a SPIFFS file into the correct flash pages 0 onwards. This takes about 1 sec, and yes if there is a powerfail during this critical 1 sec you are left with a bricked ESP that you will need to reflash using the UART bootloader.

But this risk is at the 0.001% levels, and if it happens then you are doing no more than you would have had to do anyway. However for the other 99.999% of the time, you have a usable from SPIFFS loader that works. This risk might be unacceptable for professional service provides with a diaspora of ESP devices and here the ESP32 model might be better, but this really needs a minimum 32Mbit Flash config. The bootmode that I discuss is the only one that can work with ESP8285 class configurations.

BTW, this risk is no worse than an app using SPIFFS in R/W mode. The SPIFFS code doesn't have any form of transactional recovery on page writes and there are similar powerfail time windows during output to SPIFFS which leaves you with a bricked FS.

I am not saying that I've covered all bases, but I have thought about the obvious ones. But let's get Lua 5.3 live first 😊

[raburton]

I see. Do you have copies of the readonly spiffs and uzlib that will compile baremetal that I can have a play with?

[TerryE]

@raburton, I linked to the SPIFFS repo above. Its wiki as linked tells you how to configure it. Read through the pages. But what you are really wanting here is for me to do the few days investigations and configuration that I want to defer until after I've got Lua5.3 out :smile:

The copy that we use is in app/spiffs so start with that, and override the spiffs_config.h as we do in nodemcu_spiffs.h. I would start with:

Param	Value	Comments
SPIFFS_READ_ONLY	0	We only need to read SPIFFS
SPIFFS_SINGLETON	1	Even though we only have one image, singleton hardwires the partition params as compile-time constants, but we read the partition from the PT
SPIFFS_CACHE	?	We are only reading one file serially. Need to evaluate whether enabling cache makes any material difference here
SPIFFS_IX_MAP	?	Probably 0 but need to check.
SPIFFS_TEMPORAL_FD_CACHE	0	Not needed for single file read.
SPIFFS_MAGIC	1	We use magic on our FS so this needs to be 1

I also need to validate Peter's indicative sizes on the xtensa toolchain at -O2. I am not too bothered about the RAM size, more the .text size. We'd need a new stripped down spiffs.c wrapper for the loader that uses the SPI flash read rather than the SDK wrapper.

As to uzlib_inflate.c, it's a single file. It doesn't use any SDK services and I suspect (but need to check) that it only uses the ROM C library routines. It will explode any valid RFC 1951 data stream. You just configure your uzlib_data structure and call it.

RFC 1951 uses a 32Kb window, so if we want to use standard GZIP then we will need to allocated 32Kb of RAM to this window, so long as your get_byte()(void) and recall_byte() functions handle, say 4Kb pages then we wan work within the ROM malloc/free RAM management. We would also want to buffer the put_byte() to a 4Kb buffer size, say, for writing to flash.

[raburton]

@raburton, I linked to the SPIFFS repo above. Its wiki as linked tells you how to configure it. Read through the pages. But what you are really wanting here is for me to do the few days investigations and configuration that I want to defer until after I've got Lua5.3 out

Not really, in fact the opposite, I was going to have a look at it for you. But you did say uzlib and spiffs (specifically a cut down readonly version) would compile baremetal so getting hold of these seemed like a good place to start, but the ones you mention in the nodemcu source tree do not.

I also need to validate Peter's indicative sizes on the xtensa toolchain at -O2. I am not too bothered about the RAM size, more the .text size. We'd need a new stripped down spiffs.c wrapper for the loader that uses the SPI flash read rather than the SDK wrapper.

So you don't have a baremetal compileable spiffs then? The current one is well integrated into nodemcu which is built with the SDK.

As to uzlib_inflate.c, it's a single file. It doesn't use any SDK services and I suspect (but need to check) that it only uses the ROM C library routines. It will explode any valid RFC 1951 data stream. You just configure your uzlib_data structure and call it.

Nope, doesn't compile baremetal either, at least not with my compiler, due to a lack of setjmp/longjmp support in the esp rom functions. On the other hand, I can't figure out why the author used these calls, it can probably be done differently. As you previously asserted this was compileable I assumed you'd already been here, hence asking for what you had already done before I duplicated the effort.

[TerryE]

@raburton Richard, I didn't mean to imply that the files as-is will compile to bare metal, but more that there aren't any technical impediments and the work needed to get them to compile to bare metal is small (at least compared to the sh*t that I am dealing with at the mo'): getting a baremetal ld file is a process; a fairly straightforward process, but still a process to be gone through nonetheless.

The eagle.rom.addr.v6.ld defines all (or at least the bulk) of the ROM provided entry points. Any putative link of wrapper+uzlib_inflate+spiffs* will have a set of unresolved references, and we can either resolve these by including the appropriate functions -- so long as this doesn't drag in too much other crud; or we can recode. Which ever is easier.

The author of the uzlib routines to all intents and purposes for this argument was me. I took the original LZW code plus some ideas from Paul Sokolovsky (as per attributions in header) but basically did a minimal implementation of the subset of RFC 1951 that we would need for ESP applications.

The reason that it uses longjumps is that is the "Lua way" and I am trying to be consistent with the rest of runtime's coding style. Nonetheless, recoding this with goto's to common error exit would be trivial, and looking at the objdump of libc.a, the setjmp and longjmp functions are 8 and 10 xtensa instructions with no dependencies, so this is a good example of where it is probably easier to path-in libc.a and pull the code into the bootstrap rather than recode.

[raburton]

I've had a play in between work and written a simple demo bootloader that can read a file from a spiffs, write it out to flash and boot it. Fully baremetal, nothing pulled in from sdk or any other libraries and in about 12.5k (most of which is spiffs) without playing with any options that might optimise that a bit more.

I haven't implemented any checking to see if the update is actually new or if it matches the already installed image. Not sure on best way to do this, could say always apply if the file is there in the spiffs (and allow the application to remove it on boot, so it doesn't get rewritten every time) or could CRC the existing rom to see if it matches and only flash if not. Also, not added decompression yet.

I don't know much about esp partition tables, not having played with SDK 3 so at the moment the start address for the rom and spiffs are #defined. I wonder if this allows us memory map flash across different 1mb chunks of the flash without having to intercept that mechanism (as rBoot bigflash support did). Or can you set that up in the pre_init bit on boot?

[TerryE]

Thanks Richard. I did some rough calcs and it is going to be pretty tight meeting the Sonoff challange (that is fitting in a minimal NodeMCU build with our current firmware and a SPIFFS big enough to hold the new image) unless we gz compress the image. See app/lua/lflash.c for an example of how to call uzlib_inflate() in streaming mode. We should have enough room for a 32K dictionary window allocated by the ROM malloc, but needs checking. Hopefully we can do this as a single chunk, rather than 4K pages.

See app/user/user_main.c for how the PT is declared and hooked in. user_pre_init() at L124 does the system_partition_table_regist() callback before returning control. We can just add a check here for a firmware reload RCR record, and if it exists then just bootstrap into the loader rather than setting the PT and returning to the SDK.

Note that the App can replace this PT via an updated RCR record (see app/platform/platform.c for this implementation). The RCR is just the 1st page of IROM0, and can be written to using flash nand rules for keeping effectively environment variables that can be updated and last from boot to boot, and we can use an RCR to pass the new firmware image filename to the bootstrap.

[raburton]

Thanks Richard. I did some rough calcs and it is going to be pretty tight meeting the Sonoff challange (that is fitting in a minimal NodeMCU build with our current firmware and a SPIFFS big enough to hold the new image) unless we gz compress the image.

Yes, that's what I was thinking when I said you'll need a good compression ratio.

See app/lua/lflash.c for an example of how to call uzlib_inflate()...

I've incorporated this (with quite a few modifications) into my new bootloader. It's uses a 32k dictionary so you can compress things fine with normal gzip. I've also added comparison of the crc of the image in spiffs against the installed image and if they don't match it will install the new one (after a test extract to ensure it's good) and boot it. If the ota is already installed (or there is no ota in the spiffs) it will just boot the existing rom.

See app/user/user_main.c for how the PT is declared and hooked in. user_pre_init() at L124 does the system_partition_table_regist() callback...

I've not gone this route for my demo, I'd much rather run the bootloader as a separate application and chain load the user application. If you strongly feel you want to do it that way I'm sure you'll be able to take this code and use it with minimal modification. Also, not using this SDK myself, I haven't gone into the partition stuff but I have left a function for getting this info (at the moment returns pre-set values from the header file) which is an easy point to add partition detection if needed.

The reason that it uses longjumps is that is the "Lua way" and I am trying to be consistent with the rest of runtime's coding style. Nonetheless, recoding this with goto's to common error exit would be trivial

It was only used it once and, as far as I could tell, it was entirely surplus to requirements. Moving one small block of code allowed it to be eliminated altogether. I spent a while trying to work out if it was doing something I didn't understand, but I really don't think it was (and it seems to work fine after my modification).

Code is here: https://github.com/raburton/sboot

[TerryE]

I've incorporated this (with quite a few modifications) into my new bootloader.

Great. Part of my reason for reworking and rebundling this UZlib library was to make it a simple IoT includable. The Espressif esptool flasher miniz.c is utterly unmaintainable / adaptable IMO. The UZLIB deflator omits the (Sec 3.2.7) Compression with dynamic Huffman codes because this is complex and just isn't doable in ESP8266 resource limits. However the inflator is a full RFC 1951 implementation, so as you suggest using a standard gzip library / utility on the host and the USLIB inflate with a 32Kb dictionary is a good performance compromise. I did some rough benchmarks and using a dynamic encoding scheme gains perhaps an extra 15% compression for firmware image. Not to be sneezed at. The overhead of decompression is more than compensated by the reduced SPIFFS read time as well.

The flash erase time is really quite long, so another optimisation that I've done in my LFS loader is to check for a page being all-ones and if so skipping the erase. In practice the page is either all-ones in which case the erase isn't needed or it fails with inequality within a few compares.

I had thought about adding a similar optimisation for normal writes so the the page erase / write would be skipped if the existing page already matched. This way we could use the ld file to place the startup trampoline at the head of the image so that it would in practice be a boot invariant and allow powerfail retry.

Anyway, thanks for your support, and I'll do a decent review once I've got this Lua 5.3 release out of the way.

PS. I took a quite scan. It looks really promising. :smile: Thanks again for the work.

[raburton]

The flash erase time is really quite long, so another optimisation that I've done in my LFS loader is to check for a page being all-ones and if so skipping the erase. In practice the page is either all-ones in which case the erase isn't needed or it fails with inequality within a few compares.

To give you an idea of a test I ran: image file of 405,477 bytes, compressed to 340,406. Normal startup (mounting spiffs, reading crc and length from .gz file, reading and and calculating crc of the installed rom and starting normal boot) took less than a second. Possibly quite a bit less - I was only timing with my watch! Modifying a byte to requires an update on boot (mounting spiffs, reading crc and length from .gz file, reading and and calculating crc of the installed rom, performing test decompression to ram, erasing sectors, doing real decompression to flash and then booting new image) took about 10 seconds. Didn't break that down into which bits took the time e.g. how long decompression to ram took vs decompression to flash (including the erase), etc. If you think it's the erase that's the worse part then reading first and seeing if it's needed could help. I don't know how likely you are to find a lot of pages that don't need to be erased when replacing an existing image though.

I had thought about adding a similar optimisation for normal writes so the the page erase / write would be skipped if the existing page already matched.

I can see there might be more to gain here. Although I haven't tried binary diffing roms before to see how much they change. If you build new roms from the same sdk they may look pretty similar, but it wouldn't take a big change much to displace everything slightly in the rom and need a full erase & write.

This way we could use the ld file to place the startup trampoline at the head of the image so that it would in practice be a boot invariant and allow powerfail retry.

Or just go the separate bootloader route, then you're never trying to update this bit of the flash. Now I see it can take 10 seconds to write the new image that's a much bigger window to have a fault in that'd you'd want to be able to recover from. Simpler and more clearly defined boundaries between function too and should be less work to integrate, just drop some partition reading code into sBoot and away you go.

[pawel-sw]

Separate bootloader route is a good idea, I was using rBoot for last 2 years quite extensively (too lazy to connect rs232) and the failures happen for me quite often. I'm using drag-drop web UI via HTTP to upload firmware directly to flash.

Most often failures were:

1. Background process running on timer interrupts upload or causes out of memory
2. WiFi problems
3. power supply/cables get disconnected during upload
4. EMI from nearby relay causing reset (although this one was most likely due to improper reset)

So far rBoot never let me brick my ESP.

HTTP update takes about 11 seconds to upgrade vs 7 over serial which from my perspective is negligible. The new method will add 3 seconds to upload the file to spiffs which again is negligible. I don't think we need to focus on write/erase optimisation, I did a test where 1 line change generates binary with changes in all blocks (I assume it is because of all offsets to static data which are at the end of irom0 used by l32r have to point to a different location).

Also, be careful when compiling NodeMCU at a different address than 0x40210000, I've tried merging rBoot with NodeMCU 3.0.0 with little success I think the firmware partition is hardcoded somewhere at 0x40210000 as it only works if I upload rboot firmware at 0x4020FFF0 or 0x4030FFF0.

[TerryE]

@s-pw Pawel, thanks for the comments. The 0x40210000 issue is because SDK introduced an application specified partition table which assumes this address for the IROM0 segment. Also you can't use the 3 4K pages starting at offset 0x0D000 as these are the system pages. You would need to change the PT setup code in user_main.c to reflect any allocation changes here.

Also as Richard says, this new version will use gzipped images which drop the firmware image sizes by ~ 20%.

I hope to integrate this feature around the new year.