Building an Espressif OTA (user{1,2}.bin) compatible Tasmota FW image

[mirko]

This is slightly off-topic, however -- in case my poor google skills didn't leave me in the lurch and indeed nobody managed to do so before -- might ease flashing the Tasmota supported itead/sonoff devices without HW/serial access.

I took a closer look at the original firmware, reversed engineered a few pieces and implemented a basic server which can communicate with the Sonoff devices (basically a subset of what the amazon services are doing for itead). Goal is to use the original internal upgrade mechanism to flash custom firmware onto the devices without opening them, attaching a serial cable, potential soldering, pulling down GPIO0, you know the game...

By now I managed to trick a Sonoff device to download a custom FW from servers under my control, passing the verification, and eventually flashing the image. That's some fair progress, however the device doesn't boot.

My first naive try consisted of providing a sonoff.ino.bin instead of the original upgrade file the device would usually fetch. Then I noticed, the original upgrade image doesn't contain the bootloader section, so I chopped off the first 0x1000 bytes. Still, the device doesn't boot.

That could mean that a) the original bootloader can't load the tasmota firmware (e.g. tasmota requires a modified bootloader to be loaded appropriately)? b) the internal upgrade mechanism expects a different image format

After some research I figured the original FW uses the Espressif OTA functionality. This apparently makes use of some Ping-Pong, meaning, it splits the usable flash and always flashes the inactive part. That also matches with the requests for files named "user1.bin" and/or "user2.bin".

Those -- the original upgrade files -- start with (HEX): EA 04 00 01 04 00 10 40 The Tasmota built starts with (HEX at offset 0x1000): E9 04 02 00 04 00 10 40

As you can see, the first 4 Bytes don't match, while the second 4 Bytes do. The Tasmota image header matches the Espressif documentation[1] about how an image should like (always starting with E9).

I couldn't find any documentation on what the image header in the original upgrade images is supposed to represent.

Long story short: If case b) applies ("the internal upgrade mechanism expects a different image format"), what is the that image format / header and (how) could build I build a Tasmota FW matching that image format criteria? If a) applies ("the original bootloader can't load the tasmota firmware"), can we workaround that somehow, at least for an intermediate image providing the Tasmota OTA functionality?

[1]http://www.espressif.com/sites/default/files/esp8266-sdk_application_note_firmware_download_protocol_en.pdf

[khcnz]

The world is full of funny coincidences... after sometime thinking about this i had finally started looking into this last night as well.

I presume you are using a little DNS trickery and setting up a websocket which issues the update command (along with custom url). One thing to note is that i don't believe with standard tools you can update the bootloader OTA (but maybe you can by directly overwriting the flash... it wouldn't be 'safe' from possible corruption - but in this case the only thing that could happen is to have to flash it via serial which is where we have come from anyway).

I think the first question to determine is which bootloader they are using - using esptool i believe you can dump flash from an arbitrary memory location maybe we can get some clues on their bootloader.

PS: The latest firmware that i sniffed an upgrade to (2.0.2) seems to use SSL end to end for the websockets so this could close this avenue in the future if they start shipping with newer firmware

PPS: From sniffing it would also be possible to maintain compatibility with the ewelink app infrastructure so devices could be controlled by both mqtt and the ewelink - not sure if that would be usefull for anyone..?

[khcnz]

http://esp8266-re.foogod.com/wiki/SPI_Flash_Format#0xea_Header

Seems its a newer boot mode format (boot mode 1 vs boot mode 2)

[khcnz]

also of interest https://richard.burtons.org/2015/05/17/decompiling-the-esp8266-boot-loader-v1-3b3/

[mirko]

Glad to hear somebody else is also looking into this! The hint towards the 2 image formats explains a lot - unfortunately it doesn't provide an easy solution to the issue. From analyzing the original firmware there's no (proper) way of flashing the bootloader - only the "user1" and "user2" partitions - so flashing the bootloader from within the internal original upgrade procedure doesn't seem feasible. So, if I'm understanding correctly, Tasmota images use the old image format (0xE9), including an bootloader which supports the old image type, while the original FW uses a bootloader which supports the new image format? If correct, I guess the way to go would be to build (minimal) Tasmota images with the new 0xEA header(?)

Update: it seems esptool.py as well as the arduino esptool only create 0xE9 images. :/

[mirko]

Wohoo! I just managed to flash and boot a tasmota firmware image onto a sealed Sonoff Basic! Tutorial and code will follow.

[khcnz]

(Assuming we can find a toolchain that support building the new format) I think you could maybe create a very minimal 0xEA firmware whose sole purpose is to flash the bootloader to the tasmota and enable the second stage firmware flash - however still dependent on being able to flash over the current bootloader while its running... my knowledge gets very slim here but from some reading I think the bootloader is loaded in ram... so it could be possible to flash while its running (maybe....).

[khcnz]

Edit - wow! nice! A wiki page would probably be the best place

Assuming you managed to compile tasmota with 0xEA bootloader I also guess this means we could retain backwards compatibility with the existing firmware for those who want/need to roll back?

[mirko]

Yes, I didn't touch the bootloader, just modified the image.

[davidelang]

please double-check that you can do an OTA upgrade from tasmota to a new tasmota image after this.

[mirko]

please double-check that you can do an OTA upgrade from tasmota to a new tasmota image after this.

Could you elaborate your concerns?

I'm still struggling with forcing the Sonoff to flash our custom image to the user1 partition, as -- depending what part is (in)active -- it might get flashed to user2 which is bad for several reasons. My first thought was to figure out which image the device asks for and provide a faulty image to get the device booting from the other partition, however I noticed quite quickly that the only faulty thing in this logic is me:

When the currently running system is stored on user1 (user1 == active), then we'd flash user2. Providing a faulty image only result in the bootloader not being able to boot user2 and again boot into user1.

UPDATE:

The only way I come up with right now is, in case the device asks for the user2 image (meaning user1 == active), to override the internal data segment, which comes right after the user2 one, as the internal data section stores the information for the bootloader which partition is the currently active one. But that sounds rather nasty and requires us to know the exact flash size to calculate the exact extra padding.

[davidelang]

you have been able to get tasmota running from the base build, but by not replacing the bootloader.

my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota)

If it's possible/required to do different update URLs for the two 'slots' in the router, that may not be a bad thing, we are getting to the point where size limits are pushing us to the need to do a 'two step' upgrade (first to a minimal build, then to a full build)

[mirko]

my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota)

I see, though I figured tasmota also flashes the bootloader, doesn't it? If so, then -- once tasmota is running (even with the original bootloader) -- it would OTA-update the whole flash starting from 0x0 and therewith also replace the bootloader, wouldn't it? Or does tasmota use the bootloader to perform the upgrade? That would indeed be an issue then.

If it's possible/required to do different update URLs for the two 'slots' in the router, that may not be a bad thing, we are getting to the point where size limits are pushing us to the need to do a 'two step' upgrade (first to a minimal build, then to a full build)

Good to know! That will definitely make certain things easier. However the current problem is, that the tasmota build (with the 0xEA header) only boots when being flashed as user1.bin. When being flashed as user2.bin it doesn't. Taking a closer look at the original upgrade images user1.bin and user2.bin also slightly differ. I assume the place where they'll be flashed onto flash is somehow encoded and read/used by the bootloader.

[davidelang]

On Wed, 24 May 2017, Mirko Vogt wrote:

my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota)

I see, though I figured tasmota also flashes the bootloader, doesn't it? If so, then -- once tasmota is running (even with the original bootloader) -- it would OTA-updates the whole flash starting from 0x0 and therewith also replaces the bootloader, wouldn't it? Or does tasmota use the bootloader to perform the upgrade? That would indeed be an issue then.

I'm not sure. The max size you can OTA on tasmota is 1/2 (memory size - bootloader) so I think it doesn't replace the bootloader with an OTA upgrade. When arendst is back he can comment on this with a lot more knowledge. I'm just raising potential issues in the hope that you can check them and make this rock solid.

David Lang k

[khcnz]

It's hard to say without knowing a little more about the process you are using to modify your tasmota image to boot?

I presume you have the user1.1024 and user2.1024 images from itead? When you diff them there are considerable differences (starting at the simple like byte 4 is 01 or 02 depending on which file).

I would guess that itead are using the espressif OTA mechanism - you can see more here http://www.espressif.com/sites/default/files/99c-esp8266_ota_upgrade_en_v1.6.pdf and it describes how to create the two versions aside from the bye mentioned above I believe memory addresses differ ti think..?

[khcnz]

You may also want to look at the code in the Arduino ESP8266 updater https://github.com/esp8266/Arduino/blob/master/cores/esp8266/Updater.cpp - may give some hints

[mirko]

It's hard to say without knowing a little more about the process you are using to modify your tasmota image to boot?

I'm sorry, I didn't mean to beat around the bush. esptool.py can indeed create v2 images, although that's undocumented. From the source I figured that the "--version" parameter -- while usually (and according to the docs) printing the version string of the tool -- in combination with "elf2image" accepts an parameter, specifying the version of the image to be created.

tl;dr: < esptool.py elf2image --version 2 sonoff.ino.elf > - sonoff.ino.elf in this case is the tasmota image created by the Arduino frontend.

I would guess that itead are using the espressif OTA mechanism

Yes, according to my image analysis they most certainly are.

you can see more here http://www.espressif.com/sites/default/files/99c-esp8266_ota_upgrade_en_v1.6.pdf and it describes how to create the two versions aside from the bye mentioned above I believe memory addresses differ ti think..?

Oh, interesting! So far I only stumbled over documents from Espressif describing how to use their SDK to build user1.bin and user2.bin which wasn't very helpful. The document you mentioned is way more detailed and helpful, let's see what to make out of it.

EDIT: The most interesting part of the doc probably is:

user1.bin and user2.bin are same software placed to different regions of flash. The only difference is address mapping on flash.

However it doesn't get into detail.. will now try my luck on the OTA source

[khcnz]

Great! One other thing that was confusing me that may help you... based on my reading of the Arduino IDE Uploader.cpp Arduino OTA mechanism (that tasmota uses) doesn't seem to ever flash OTA user 1 image... only ever user 2. User1 is left for fallback as far as I can tell (see how it calculates the start address of where to write the flash to. Thats why the fact that the tasmota/arduino image has the first 0x1000 as the bootloader is irrelevant. its written to a space that isn't used (but according to the link above its kept for symmetry reasons).

This differs from espressif as far as i can tell where they write alternatively between the two - you can tell which one is currently executing with the call system_upgrade_userbin_check

[khcnz]

Check this out - tasmota only uses linker script 1 https://github.com/arendst/Sonoff-Tasmota/tree/master/arduino/version%202.3.0/tools/sdk/ld

Look at this example of a make script building 1 & 2 images https://github.com/jeelabs/esp-link/blob/master/Makefile

I think we just need to have a secondary ld script

[khcnz]

One last link :) https://github.com/espressif/ESP8266_RTOS_SDK/tree/master/ld you can compare eagle.app.v6.new.1024.app1.ld and eagle.app.v6.new.1024.app2.ld

I think if you changed the tasmota over-ride and change irom0_0_seg from 0x40201010 to 0x40281010 it will work

[arendst]

With these user1 and user2 images I see two concerns:

• What happens to the arduino EEPROM flash area at address fb000? Is it left untouched by the max size image?
• How about the nice arduino feature of being able to OTA upload a larger image than half the flash size as it uses dynamic image loading

Just some thoughts during holiday...

[mirko]

What happens to the arduino EEPROM flash area at address fb000? Is it left untouched by the max size image?

Well, I'd say (for now) this method only allows flashing images < 0xfb000−0x81000 = 488KB which at least a minimal tasmota image shouldn't exceed (or would it?). From there we could second-stage flash a normal image, then also incl. the bootloader. Does that make sense?

How about the nice arduino feature of being able to OTA upload a larger image than half the flash size as it uses dynamic image loading

Honestly, I've absolutely no idea as this is my first arduino project I kinda involve myself and frankly this arduino world doesn't appear to be mine so I'm not sure I'd be of any help here..

Just some thoughts during holiday...

Enjoy your holidays! When are you gonna be back? And where are you? And how is it? :)

[khcnz]

@mirko How did you go with building a user2 image?

@arendst I can't find any info on dynamic sized arduino images but I have done a lot more digging and finally have my head around how the boot rom, bootloader and ota is working.

1. Boot Rom - this is flashed into the actual ESP (not the flash chip). Used to setup system / spi flash etc and then run normal bootloader. Also takes care of the process for flashing via UART when put into flash mode. The boot rom can not be updated but can be dumped out as it is mapped to a flash memory address Reverse engineering info here

2. Bootloader - this is normally flashed to 0x0 - espressif and arduino both have their own bootloaders and both work very differently and have have quite strict flash layouts. The bootloader is essentially responsible for any custom setup and also determining the entry point to the 'real' application. See more below

3. Espressif Bootloader - Splits flash essentially in 1/2 - an image can be flashed to either half and the bootloader is responsible for determining which to execute. 'Safety' is created by writing to the other 1/2 of flash from the one being used and then switching over at next boot.

   • Write up
   • Code

4. Arduino OTA - Works quite differently - still splits the flash in 1/2 but OTA updates always save the new sketch to the second half of flash. The bootloader then copies the newsketch from the second half of flash back to 0x0 on next boot.

   • Write up
   • Code

So.. in conclusion

1. It seems should be possible for sure to create both user1/user2 images that the espressif bootloader will work with - just need to check out how the layout is flashed (especially since the images from iteads servers don't seem to have the bootloader AFAIK - just the application code. Perhaps these are even just stub images that don't do much but allow for a secondary image to be loaded? Flashed to match the arduino bootloader?

2. The the arduino bootloader is pretty restrictive if we want to include larger than 1/2 of flash size images (and when i say restrictive i mean not possible).

3. There is a 3rd party esp8266 bootloader rboot that seems to fit the bill and allows arbitrary # of roms and rom sizes. I would suggest we would have 3 binaries

   • Bootloader
   • Very very small (smaller than minimal) upgrade rom - would need to be able to boot into this rom and accept a new image. Whats the best way to do this while still maintaining flexibility - download from a url? webpage? The more options we add the less space is available for the main application.
   • Normal application / sketch

Obviously depending on how small the small upgrade rom could get - we could have a lot more flash available for the main application

Thoughts?

(edited images to be inline)

[davidelang]

the #define BE_MINIMAL is intended to do this. It builds an image that has almost all features disabled, but still has the wifi and webserver functionality so that you can tweak configs and download/install a new image.

If it's not small enough, we would need to figure out what's eating the space and can be disabled.

[khcnz]

From memory the minimal still includes MQTT and there is a ton of config upgrade logic that wouldn't need to exist, and the webserver code is massive as well. Even just the base arduino and espressif libraries add a ton.

Clearly the smaller it can be the better - but the smaller it is made the less ways we could support upgrade (the smallest way would probably only to support connecting to a defined wifi network and pulling from a defined url). Of course then you get into questions like dns/mdns, HTTPS, static ips etc etc.

[davidelang]

there isn't actually a way right now to not compile in the mqtt code.

I think the webserver is still needed to be able to direct an upgrade and configure things like network connection.

As long as the minimal build + real build < flash space (- bootloader), it should be good enough

It should also be possible for the firmware to overwrite the bootloader to conver it from the stock sonoff firmware to the one that tasmota has been using.

[mirko]

https://github.com/espressif/ESP8266_RTOS_SDK/tree/master/ld you can compare eagle.app.v6.new.1024.app1.ld and eagle.app.v6.new.1024.app2.ld I think if you changed the tasmota over-ride and change irom0_0_seg from 0x40201010 to 0x40281010 it will work

This indeed sounded like the hint, however the images I managed to create (user1 = without that change, user2 = with that change) only differ in what I think is the build timestamp. Nothing like the diff between the original user1/user2 upgrade images :/

[mirko]

the #define BE_MINIMAL is intended to do this. It builds an image that has almost all features disabled, but still has the wifi and webserver functionality so that you can tweak configs and download/install a new image.

BTW, that fails for me (current master) with:

"/home/mirko/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/1.20.0-26-gb404fb9-2/bin/xtensa-lx106-elf-g++" -D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/lwip/include" "-I/tmp/arduino_build_518542/core" -c -w -Os -g -mlongcalls -mtext-section-literals -fno-exceptions -fno-rtti -falign-functions=4 -std=c++11 -MMD -ffunction-sections -fdata-sections -DF_CPU=80000000L -DLWIP_OPEN_SRC   -DARDUINO=10803 -DARDUINO_ESP8266_ESP01 -DARDUINO_ARCH_ESP8266 -DARDUINO_BOARD="ESP8266_ESP01"  -DESP8266 "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/variants/generic" "-I/home/mirko/Arduino/libraries/PubSubClient/src" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/Ticker" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/ESP8266WiFi/src" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/ESP8266HTTPClient/src" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/ESP8266httpUpdate/src" "-I/home/mirko/Arduino/libraries/ArduinoJson/src" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/ESP8266WebServer/src" "-I/home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/libraries/DNSServer/src" "/tmp/arduino_build_518542/sketch/sonoff.ino.cpp" -o "/tmp/arduino_build_518542/sketch/sonoff.ino.cpp.o"
/home/mirko/src/Sonoff-Tasmota.git/sonoff/sonoff.ino: In function 'void GPIO_init()':
sonoff:2384: error: 'dht_setup' was not declared in this scope
         if (dht_setup(i, mpin)) {
                              ^
[..]
exit status 1
'dht_setup' was not declared in this scope

[davidelang]

that section should be wrapped with #ifdef USE_DHT

a patch similar to:

diff --git a/sonoff/sonoff.ino b/sonoff/sonoff.ino index c022965..203d60a 100644 --- a/sonoff/sonoff.ino +++ b/sonoff/sonoff.ino @@ -2380,6 +2380,7 @@ void GPIO_init() led_inverted[mpin - GPIO_LED1_INV] = 1; mpin -= 4; } +#ifdef USE_DHT else if ((mpin >= GPIO_DHT11) && (mpin <= GPIO_DHT22)) { if (dht_setup(i, mpin)) { dht_flg = 1; @@ -2388,6 +2389,7 @@ void GPIO_init() mpin = 0; } } +#endif } if (mpin) { pin[mpin] = i;

[mirko]

BTW, this is the current state of my implementation instructing a sonoff device to fetch and flash a custom image from an arbitrary (reachable) server: https://pb.nanl.de/show.php?id=2765&hash=14959135

[khcnz]

This indeed sounded like the hint, however the images I managed to create (user1 = without that change, user2 = with that change) only differ in what I think is the build timestamp. Nothing like the diff between the original user1/user2 upgrade images :/

Are you sure you changed the correct file? I just rebuilt mine with the change and the files were very different (in a similar way to the difference between itead user 1 and user2 files).

Not sure what platform you are running on but it is not this file Sonoff-Tasmota\arduino\version 2.3.0\tools\sdk\ld\ but the one in ArduinoData\packages\esp8266\hardware\esp8266\2.3.0\tools\sdk\ld that needs modification.

[mirko]

Are you sure you changed the correct file?

I thought so...

Not sure what platform you are running on but it is not this file Sonoff-Tasmota\arduino\version 2.3.0\tools\sdk\ld\

Oh...

ArduinoData\packages\esp8266\hardware\esp8266\2.3.0\tools\sdk\ld[eagle.flash.1m64.ld]

That actually did the job!

In my case it's: /home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/ld/eagle.flash.1m64.ld (Linux)

IT WORKS!

I assume /home/mirko/.arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/ld/eagle.flash.1m64.ld is the linker script coming from the esp8266 library? So the tasmota linker scripts are not used at all when using the Arduino environment?

[khcnz]

When using the arduino ide you are supposed to replace the inbuilt arduino linker script with the one inside tasmota - the only difference is theo has provided a non SPIFFS version (which arduino ide doesn't have out of the box) - there are some instructions here https://github.com/arendst/Sonoff-Tasmota/wiki/Prerequisite

[khcnz]

And by "it works" I assume you mean you can now flash both User1 and User2 images "OTA" from stock itead firmware and they run. I presume you are still stripping off the first 0x1000 bytes from the build that arduino does (to remove the bootloader) - since the itead/espressif ota isn't expecting the bootloader.

Ideally we can now find a way to bootstrap replacement of the bootloader as well as I think right now OTA won't work from the booted tasmota image since the OTA process works differently.

[mirko]

And by "it works" I assume you mean you can now flash both User1 and User2 images "OTA" from stock itead firmware and they run.

Correct.

I presume you are still stripping off the first 0x1000 bytes from the build

As I convert the ELF images to the v2 image format -- which don't have the bootloader included -- I don't have to chop off the first 0x1000 from those.

[mirko]

Ideally we can now find a way to bootstrap replacement of the bootloader as well as I think right now OTA won't work from the booted tasmota image since the OTA process works differently.

Indeed it doesn't work. But right now mainly because the image (I assume) gets now flashed to user1 (as we're running our custom tasmota image right now from user2 as we flashed from the original FW from user1) however the bootloader still tries to boot from user2.

As the tasmota image is a bit bigger -- when being written to user1 -- it also (again, I only assume) overrides parts of the user2 partition. So booting user2 fails after the update.

This would also imply the bootloader doesn't get flashed indeed via OTA since I'd expect the tasmota bootloader loading from 0x1000 (user1).

So we either need to replace the bootloader (which would be best I guess as then the device would be in a state as it would be after being flashed via serial) or tell the original bootloader to switch back to user1 after successful flashing (which would imply using and modifying the system information partition (last 16KB)).

[khcnz]

From my reading of the Arduino OTA and Espressif OTA code I think the problem you will be running into is the the Arduino OTA (which is running once you flash) is expecting a bootloader header.

Checkout packages\esp8266\hardware\esp8266\2.3.0\cores\esp8266 Arduino saves the following info to RTC which the Arduino bootrom tries to pickup

if (_command == U_FLASH) { eboot_command ebcmd; ebcmd.action = ACTION_COPY_RAW; ebcmd.args[0] = _startAddress; ebcmd.args[1] = 0x00000; ebcmd.args[2] = _size; eboot_command_write(&ebcmd);

So write out the data from _StartAddress on flash to 0x0 with _size (so obviously the bootloader must be replaced)

but the Espressif bootloader tries to run from the following for user2 0x4010800c((((offset + 4) >> 1) << 12) + 0x1000);

Which i haven't math'd but pretty sure that its saying find 1/2 the flash size then skip the first 0x1000 bytes - which is going to put it 0x1000 bytes into your image - rather than the start of it.

[khcnz]

Actually edit that - I think a closer reading of Update.cpp and the Arduino OTA flashlayout - it is writing the Update to a block aligned flash address just after the end of the current sketch - note 1/2 way...

Suggest if you updated Updater.cpp (need to triple check this is what is actually being used by tasmota) so that the image is written out to 0x4010800c((((offset + 4) >> 1) << 12) + 0x1000); it would work - but as you say you may end up hitting into other flash regions that the Espressif code uses depending on code size.

[mirko]

I wonder if it makes sense at all to get that into the tasmota project or if it would be better to create a very own espressif-ota->arduino bootstrap project which compiling would result in 2 minimal images (user1.bin / user2.bin), having the v2 image format right away, and providing Arduino OTA functionality (incl. support for flashing the bootloader) for properly switching to custom FW (e.g. tasmota)

Or we add code to the tasmota project (=OTA code) checking which bootloader is used and respectively replace it or not. The image the Tasmota/Arduino OTA code expects has a bootloader included anyway and apparently just gets chopped off after being transmitted via HTTP.

[khcnz]

Agree I think a standalone would probably be best - working on the assumption you can't flash what you are running I think we need three parts

1. User1 image - ony purpose is to kick-off the user2 image update using the Espressif OTA functionality as we can't overwrite bootloader while running as user1 image
2. User2 image - Download "User3" image and flash directly using SPI erase/write to 0x0. Would need to manipulate the flash to change which image is booted next time.
3. An Arduino IDE build (can be tasmota).

[mirko]

I put the script on github now: https://github.com/mirko/SonOTA and published the progress and findings in a blog post: http://blog.nanl.de/2017/05/sonota-flashing-itead-sonoff-devices-via-original-ota-mechanism/

I'm happy about criticism / feedback including tips pointing out missing / insufficient credits.

[khcnz]

Nice writeup! I might just get some time this week to do some coding rather than just google kung-fu.

[khcnz]

Some additional info that could be helpful for creating 'dummy' OTA builds which i hope to look at soon. From dumping out the firmware before/after the ewelink app stores the wifi credentials you enter at 1) SSID: 0x79000 -> 0x7901F (null terminated / padded) - Up to 30 chars +\0 2) Password: 0x79020 -> 0x79066 (null terminated / padded) - Up to 70 chars including \0

[khcnz]

I have a custom little flashing image now that can install OTA and then reflash the other 1/2 of flash not currently running and force the reboot and switch to the new image. But working on making it a bit simplier (from a user perspective) but a bit more complicated from a code perspective.

Pretty sure I can make this work with one image that can run as an Espressif image on the second ROM slot (and always get there even if user1 is requested) - then flash on the Arduino bootloader and then copy the running image on rom 2 back to slot 1 while dynamically changing the header so that the arduino bootloader can start it up... After that you would just need to connect to the a webpage and flash tasmota.

[mirko]

I have a custom little flashing image now that can install OTA and then reflash the other 1/2 of flash not currently running and force the reboot and switch to the new image. But working on making it a bit simplier (from a user perspective) but a bit more complicated from a code perspective. Pretty sure I can make this work with one image that can run as an Espressif image on the second ROM slot (and always get there even if user1 is requested) - then flash on the Arduino bootloader and then copy the running image on rom 2 back to slot 1 while dynamically changing the header so that the arduino bootloader can start it up... After that you would just need to connect to the a webpage and flash tasmota.

Yeah, that sounds like great process! Thanks for your effort on this!

[khcnz]

Getting pretty close to have this working end to end will probably finish over weekend. Currently have a standalone version and thinking it will easily integrate into tasmota with #ifdefs so that its a one OTA rom download from stock to running tasmota.

[cjcharles0]

Been following this discussion in the background and really interested/exciting to see the progress. I would ideally like to integrate it with different firmware from Tasmota so using some #ifdefs would certainly be a great implementation, though I guess this will be more complex than having a first flash file which is universal (and just enables the update page), followed by flashing the actual arduino bin file...

[mirko]

I also just pushed a new version of the SonOTA script which now can also provision the device (without the need of the proprietary app). This work is based on the research of Jan Almeroth ( https://wiki.almeroth.com/doku.php?id=projects:sonoff#pairing ). Neat side effects are that while provisioning we can now also specify the serving host, which means

• we do not need to spoof / redirect DNS requests
• it doesn't need to run as root (as we can also specify the port and don't need to listen on 443) anymore.

[khcnz]

Nice update mirko - this is very very good!

On my end I have mixed news Dynamic Self Patching idea - Download to User2, rewrite bootloader and copy over application to slot 1. Bootloader successfully replaced and reboot works fine - HOWEVER the symbols inside irom.text now sitting in the first half of flash still resolve to memory address mapped in the 2nd half of the flash partition. Everything works until you can erase the second half of flash (and of course therefore can't do another following OTA.

By comparing the differences between user1/user2 bin files I have worked out how I can rewrite the memory addresses on the fly with some heuristics and this works well for a standalone example - however fails with something as complex as Tasmota.

So may have to go back to the 3 file idea 1) User 1 Espressif Image which Downloads User2 Espressif Image 2) User 2 Espressif Image which downloads Tasmota and rewrites to 0x0 3) Tasmota

The 3 file version isn't quite as neat but does work - i'll guess I'll have to put aside the more complicated version until I can find some references on the layout inside the irom portion. Should be able to post some code tomorrow of the 3 step version.

PS: If anyone knows how to parse the irom section and remap the symbols please get in touch. Remapping is easy just involves subtracting 0x80000 from memory addresses of all items.. but knowing which values are memory addresses and which are just values is proving hard

[mirko]

If anyone knows how to parse the irom section and remap the symbols please get in touch

I dropped Richard Burtons[1] an email pointing to that ticket - he might have the insights and be willing to provide some help on this.

[1]https://richard.burtons.org/?s=ESP8266

[raburton]

You can't remap the application at runtime - it isn't simply a case of the base address being stored in the header, the elf is linked with that address baked in in ways you will not be able to understand / adjust in the completed binary. You need to have two separate roms - one for the lower 1/2 and one for the upper 1/2. Yes, that isn't as neat but it's no big deal. If there had been a bigger flash than 8mbita on the device you could have used a single rom, by spacing them in separate mbs of flash each mb will be mapped to the same base address in memory. Of course they have used the smallest flash they can for cost purposes. Luckily it's trivial to create both versions at link time and then when you ota you just need to request the correct one file be downloaded and flashed.