Experimenting that resulted in repeat successful reverting of Tasmota to Shelly via OTA.

[pandabear41]

I have been playing around reverting Tasmota back to the native Shelly mgos firmware via OTA. I think I might of found something. Most of the recovery Shelly FW files are 2MB. I was checking out the issue on the arendst/Tasmota#8436 and looking at the comments. In the comment listed here, the 2MB flash layout was linked here and I noticed something, it has two applications listed in the flash. Once again I was stupidly playing around with flashing my Shelly 2.5, I decided to backup the first 1MB of the native Shelly flash using USB to Serial adapter.

I repeated this steps below 5 times, all successful.

Pre-Steps (To generate a 1MB recovery image)

1. Flash the Shelly with the recovery firmware from this url with USB-to-Serial adapter. https://www.shelly-support.eu/filebase/index.php?entry-download/112-shelly2-5-recovery-firmware-1-5-10/&fileID=152
2. Backup first 1MB of Shelly flash using USB-to-Serial adapter. You can use "esptool.py" with the following command python esptool.py -b 115200 --port COM4 read_flash 0x000000 0x100000 shellyBackup.bin. You can also use Tasmotizer backup checkbox as it only save 1MB of flash.
3. Compress the 1MB .bin file using gzip.

Steps

1. Flash mgos -> Tasmota using http://shellyip/ota?url=http://dl.dasker.eu/firmware/mg2tasmota-Shelly25.zip
2. Connect Tasmota to wifi.
3. Flash the Tasmota Minimal via OTA.
4. Flash the compressed 1MB dump via OTA on Tasmota. If the flash fails you might have to run the following in Tasmota SetOption78 1 console and reboot before trying again. This takes a few mins to complete.

Notes

After reverting Shelly firmware version will be ????, this is fixed after updating. Tried with more than one Shelly 2.5.

Warnings

Only tested with Shelly 2.5. This doesn't work if you do a Reset command to clear flash in Tasmota. During testing only the Shelly recovery firmware image will work. Tried V1.8 and others and got a boot loop. Also during testing if you don't use the mgos-to-tasmota via Shelly OTA it won't work. You will get a boot loop if you don't use mgos-to-tasmota. This experiment was a result of drunk coding, so try at your own risk.

Ideas on why this works

mgos-to-tasmota copies app0 to app1. Then it puts Tasmota around first 1MB of memory and doesn't clear entire flash, so I am guessing "rf cal" and "sys params" don't get cleared.

[yaourdt]

This is a very clever idea! I didn't think of compressing the image.

I - so far - tried to build a Mongoose OS minimal image that would fit within the 512K limit, and thanks to @rojer I'm almost there. The advantage of an uncompressed image below the 512K limit is that we have a universal way back, not just a way back from Tasmota. We could even flash it via TUYA convert etc. But if I cannot get this running within the next few days, and with your permission, I'd like to provide binaries as you described above.

[pandabear41]

That would be nice to get a minimal Mongoose OS. The problem that I see is that the Mongoose OS configured with 2MB flash size it puts the sys_params at the end of the 2MBs. I wonder if you can compile it with 1MB flash that way you don't have all the empty sections (which brings it over 512K with compression). I've tried flashing the full 2MB recovery flash image, the problem is even with compression it exceeds the 512K limit. Doing the method I stated above assumes you have "sys params" left from other Mongoose OS flashing. Otherwise rboot just loops.

I was thinking of just making a minimal Arduino esp8266 that just has Arduino OTA that way it can program larger images and if it has enough space I can code in custom REST or tcp commands to add in anything we want to the flash (like a sys params section) or even methods to program the Mongoose OS format.

[rojer]

sys_params location is fixed at the last 16K of flash

[yaourdt]

This is why I did not worry about the sys_params up until now: If I understand it correctly, the partition is needed by the esp chip to boot, and subsequently will be independent of the firmware we are using. So, as long as we do not overwrite the partition, we should be good. This is assuming we do not change the parameters that govern the flash size, and thus implicitly move the position where the esp is looking for its sys_params. The flash size (and other parameters such as mode / freq) are set in the boot partition header.

I'm not perfectly certain, but I think most firmwares such as Tasmota will use a fixed flash size of 1MB, so the sys_params are most likely located at 1012 kB (4kB is WiFi cal) and we can use this in first approximation. However, I've not confirmed this as of yet.

[pandabear41]

I built a tiny Arduino OTA using just example code and have been successful flashing tasmota -> arduino ota -> yaourdt/tasmota-to-mgos. Trying to use the OTA to flash a Shelly .zip, it fails after that. Serial output shows it flashes successfully. I'm not sure why it doesn't work. Have you been successful with yaourdt/tasmota-to-mgos? I also tried tasmota -> arduino ota -> Shelly 2.5 full 2mb compressed image (around 550k). It flashes successfully and boots mgos, but it never starts up completely.

@yaourdt Tasmota is fixed 1MB flash size, as I have flashed an esp8266-01 without changing anything. That explains why the 2mb sys_params from the original Shelly aren't wiped out and I am able to flash first 1MB of Shelly recovery image without any issues.

[yaourdt]

Trying to use the OTA to flash a Shelly .zip, it fails after that. Serial output shows it flashes successfully. I'm not sure why it doesn't work.

Could you send me the serial output?

I also tried tasmota -> arduino ota -> Shelly 2.5 full 2mb compressed image (around 550k). It flashes successfully and boots mgos, but it never starts up completely.

Depends on what exactly your arduino code is doing. Also, how far does it start before it fails?

[yaourdt]

@pandabear41 I managed to squish a minimal version of Mongoose OS including sys_params in a 512K firmware image: mgos512k.bin.zip

This should boot a minimal version of tasmota2mgos and you can connect to the WiFi. I'll still have to sort out some minor update issues, but we are almost there

[yaourdt]

Okay, I have it working on an nodeMCU dev board under one condition: The minimal firmware is flashed with --esp-flash-params 'dio,32m,80m' (4MB). If I flash the minimal firmware with 'dio,4m,80m' (512KB), minimal will boot just fine, but writing the update to address 0x200000 fails.

I've attached detailed logs, but I think it is due to the ESPs 1MB write window not moving up to 0x200000 when booted with param 4m.

@rojer - this is this a limitation in the ESP design itself, not Mongoose OS/rboot, am I right? If so, how could I get around that? If this is true, the only solution I can think of would be: After initial flash try to patch rboot headers and reboot - but then sys_params would be in the wrong position again, and if there is no sys_params present from a previous firmware in the right position, the device won't boot.

[yaourdt]

@pandabear41 please try this image. It should work for Shelly1, Shelly1PM, Shelly2, Shelly25, and ShellyEM under the following assumption:

1. You flashed tasmota using mg2tasmota coming from Shelly stock firmware
2. You did not wipe the flash (more precisely the sys_params partition at 0x1fc000 is intact)

If this works on your Shelly 2.5, can you please confirm?

[pandabear41]

@yaourdt Tried your image it seems to fail with an error after booting. I started with Shelly stock 1.8 -> m2tasmota -> tasmota minimal -> your image. All other than Shelly stock programmed OTA. sec 7e error just repeats forever. Power cycling the device gives no serial output and no LEDs turning on or flashing.

16:52:56 HTP: Main Menu
16:52:57 HTP: Firmware Upgrade
16:53:01 WIF: Checking connection...
16:53:02 UPL: File mgos512k-2MB.bin ...
16:53:12 UPL: Successful 524288 bytes. Restarting
16:53:12 HTP: Upload done
16:53:12 SRC: WebGui from 192.168.15.114
16:53:13 APP: Restarting

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 3456, room 16 
tail 0
chksum 0x84
csum 0x84
v0cc1cc0b
@cp:0
ld

sec 7e error
sec 7e error
sec 7e error
sec 7e error
sec 7e error

[yaourdt]

I see, sorry for that. Same image, but valid: flash2.bin.zip Could you try again, please?

[pandabear41]

Works better and I get further than I did before. Shelly tries to boot, but it fails with a new error. I started with Shelly 2.5 v1.8 -> m2tasmota -> tasmota minimal -> your image (guessing tasmota-to-mgos) -> Shelly 2.5 v1.8 (using curl -i -F filedata=@./fw.zip http://10.42.42.44/update) Looks like the mgos OTA gets confused and boots the incorrect rom section. Might need to move your app to rom1 so OTA programs rom0. Or something else is going on.

Logs (some repeated messages omitted): https://gist.github.com/pandabear41/72559faad5421e609282108cdafb0bad

[yaourdt]

Ok, I can reproduce it. This problem occurs only for Shelly firmware, not for a 'normal' Mongoose OS build, which I use for testing. Normal builds are fine; rboot is supposed to boot from rom1 after the update, so this is ok as well. The problem happens at a later stage of the firmware boot process. I'll have to research what additional steps have to be taken for a Shelly firmware to complete its boot process.

[rojer]

MISSING HWINFO is the problem here.

it needs a file called hwinfo_struct.json on the filesystem. i don't know all the rrequired fields but here's one from shelly1:

{"batch_id": 2, "model": "shelly1", "manufacturer": "Allterco", "hw_revision": "prod-2018-08", "hwinfo_ver": 1}

this file is not included in the firmware and i believe is added after production.

[yaourdt]

Thanks, @rojer that did the trick, at least on the dev-board. Let's see if this will also work in real life. New image: flash-autodetect.bin.zip

I've also included a patched version of rboot, which will move the sys_params partition at boot time, just to make sure. And it should now auto-detect the true flash size from the esp8266 chip ID.

@pandabear41 - sorry, I'll have to ask you to test it again, as I don't own a Shell 2.5

[yaourdt]

@rojer - is there a way for removing another 5248 byte from the firmware?

current config:

libs:
  - origin: https://github.com/mongoose-os-libs/wifi
  - origin: https://github.com/mongoose-os-libs/mbedtls         # remove support for crypto chip to reduce footprint
    variant: esp8266-noatca
  - origin: https://github.com/mongoose-os-libs/mongoose        # remove SSL support to reduce footprint
    variant: esp8266-nossl
  - origin: https://github.com/mongoose-os-libs/ota-http-server # local OTA update via HTTP POST

which is just slightly too much for rboot config at 0x7000:

#
# 0.5MB (4Mb) nano layout:
#
# 0x000000 - rBoot       |
# 0x007000 - rBoot cfg   |
# 0x008000 - app (450K)  | x
# 0x077000 - fs (16K)    |
# 0x07b000 - rf cal      |
# 0x07c000 - sys params  |

thanks again for your help!

[pandabear41]

@yaourdt That did the trick. It works great and I got back to Shelly native FW on my Shelly 2.5.

Logs for your reference. https://gist.github.com/pandabear41/2b7594ae66e07b9d8c79bf0c152ffc40

[rojer]

@rojer - is there a way for removing another 5248 byte from the firmware?

i've had a look and found that we were bringing in sha512 hash algo even though we aren't using it. i made some internal changes and reduced size by approx. 12K:

$ ls -l build*/objs/*.bin
-rw-r--r-- 1 rojer rojer 443520 Aug 21 22:22 build/objs/empty.bin
-rw-r--r-- 1 rojer rojer 455808 Aug 21 22:01 build.old/objs/empty.bin

please rebuild

also, don't forget to set:

build_vars:
  MGOS_MBEDTLS_ENABLE_ATCA: 0

cdefs:
  MG_ENABLE_SSL: 0

in addition to the lib variants, otherwise headers may be out of sync with binaries.

[rojer]

you can save additional 18K by removing LFS dependency here.

[yaourdt]

Sorry for taking so much time, but I think it is done and should work for both boot config addresses. Thank you very much to both of you for your help!