How to compile a ESP32 OTA firmware package

[gampam2000]

Hi,

Your project looks very interesting and I want to try it out. I recompiled Micropython with BOARD_VARIANT=OTA

but when i try:

import ota.status
ota.status.status()

I always get an oserror and esp debug says that no otadata was found. I tried to flash the firmware.bin and the individual images separately with the same effect. At the end of the build it says:

bootloader  @0x000000    19152  (   13616 remaining)
partitions  @0x008000     3072  (    1024 remaining)
application @0x010000  1685632  (  345984 remaining)
total                  1751168

The Partition table file looks like this, i have an ESP32S3 with 2MB PSRAM and 8MB Flash:

nvs,      data, nvs,     0x9000,   0x4000,
otadata,  data, ota,     0xd000,   0x2000,
phy_init, data, phy,     0xf000,   0x1000,
ota_0,    app,  ota_0,   0x10000,  0x180000,
ota_1,    app,  ota_1,   0x190000, 0x180000,
vfs,      data, fat,     0x310000, 0x600000,

The Micropython esp32 doc does not state anything other than use the BOARD_VARIANT=OTA. Do you see any obvious errors or could you point me in the right direction of what I'm doing wrong?

Thanks & best regards.

[glenn20]

Hmmm.. Im not sure what the problem is here.

It would be helpful if you could provide the error message and what board/chip you are using.

I do recommend completely erasing the flash esptool.py erase_flash before flashing an OTA firmware over an existing firmware (however, I dont think that would cause this problem).

On another note: the default partition table for BOARD_VARIANT=OTA is for a 4MB flash device, and so has smaller partitions for the micropython firmware (ota1 and ota2). It appears that your device has a partition table for a 8MB flash. If so, the bootloader also needs to be told that the flash is 8MB. Compiling with the BOARD_VARIANT option does not make it easy to combine features, eg. OTA and 8MB flash.

It may be easier to compile (or download) a regular micropython image and use my mp-image-tool-esp32 to modify the image for your device:

• mp-image-tool-esp32 -f 8MB --table ota firmware.bin

and try flashing the new image onto your device. This will also have the benefit of installing larger partitions for your firmware (which can be accommodated on an 8MB flash device).

You do need to have python 3.10 or above to use mp-image-tool-esp32 for now.

[glenn20]

Actually, I just looked more closely and your partition tables suggest your flash is 9502720 bytes long (0x310000 + 0x600000) - which is very odd. Somehow, your partition table seems to have set the length of the vfs partition incorrectly, which may be causing odd errors (but I need to see the error messages to know if that is true ;) ). mp-image-tool-esp32 u0 will report the actual flash size of your device.

[gampam2000]

Thanks to your great tool, it tells me that the firmware is compiled without OTA partitions.

mp-image-tool-esp32$ python3.10 mp-image-tool-esp32 ../micropython/ports/esp32/build-SET_SMW_S3-OTA/firmware.bin 
Opening image file: ../micropython/ports/esp32/build-SET_SMW_S3-OTA/firmware.bin... 
Chip type: esp32s3 
Flash size: 8MB 
Micropython App size: 0x19b880 bytes (1,646 KB) 
Partition table (flash size: 8MB):
# Name             Type     SubType      Offset       Size      (End)  Flags
  nvs              data     nvs          0x9000     0x6000     0xf000  0x0  (24.0 kB)
  phy_init         data     phy          0xf000     0x1000    0x10000  0x0   (4.0 kB)
  factory          app      factory     0x10000   0x1f0000   0x200000  0x0   (1.9 MB)
  vfs              data     fat        0x200000   0x600000   0x800000  0x0   (6.0 MB)

So I guess it ignores my partition table, I have configured it in the sdkconfig.board:

CONFIG_ESPTOOLPY_FLASHSIZE_4MB=
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-8MiB-ota.csv"

where does this custom partition table need to be? I have it in/micropython/port/esp32/

You where also correct that my partition table was wrong, i modified the existing 4MB OTA table for 8MB and got the offsets & sizes wrong. This should now be correct:

# Name,   Type, SubType, Offset,   Size,     Flags
nvs,      data, nvs,     0x9000,   0x4000,
otadata,  data, ota,     0xd000,   0x2000,
phy_init, data, phy,     0xf000,   0x1000,
ota_0,    app,  ota_0,   0x10000,  0x1F0000,
ota_1,    app,  ota_1,   0x200000, 0x1F0000,
vfs,      data, fat,     0x3F0000, 0x410000,

With your tool I successfully created an OTA Image. And it seems to work:

>>> import ota.status
>>> ota.status.status()
Micropython v1.23.0 has booted from partition 'ota_0'.
Will boot from partition 'ota_0' on next reboot.
The next OTA partition for update is 'ota_1'.
The / filesystem is mounted from partition 'vfs'.
Partition table:
# Name       Type     SubType      Offset       Size (bytes)
  nvs        data     nvs          0x9000     0x5000     20,480
  otadata    data     ota          0xe000     0x2000      8,192
  ota_0      app      ota_0       0x10000   0x200000  2,097,152
  ota_1      app      ota_1      0x210000   0x200000  2,097,152
  vfs        data     fat        0x410000   0x3f0000  4,128,768
OTA record: state=VALID, seq=1, crc=1195612314, valid=True
OTA record is VALID. Will be updated on next boot.
OTA record: state=UNDEFINED, seq=4294967295, crc=4294967295, valid=False
OTA record is UNDEFINED.
Next boot is 'ota_0'.

So the only question that remains is, where or how can I configure a custom partition table when compiling micropython?

[glenn20]

Excellent. Glad it works (so far).

You can just use mp-image-tool-esp32 to change the partition table after you build it... which is handy because you can just use the images downloaded from the micropython downloads page.

I see from your message above, that you seem to have created your own board definition SET_SMW_S3-OTA in the ports/esp32/boards directory. You didn't answer my question about what board/chip you are using. It is hard to give specific advice without more information. From the name of your board definition, I guess it is an ESP32S3.

You could just copy the board config for the board you use and then edit it to use the right partition table and flash size OR you could add a new board variant combining everything you need to an existing board definition.

You can make a copy of one of the existing partion*.csv files in ports/esp32 and edit it to suit.

[gampam2000]

Hi, yes with your mp-image-tool-esp32 it works. But i need to compile Micropython because I need some special drivers.

I use an ESP32-S3-WROOM-1-N8 on my board. I copied the/boards/ESP32_GENERIC_S3 to my board definition and edited the sdkconfig.board and also created the partition table: partitions-8MiB-ota.csv that is based on a copy from the default 8MiB definition.

When compiling with BOARD_VARIANT=OTA it does not create an OTA image. My guess is either BOARD_VARIANT=OTA is ignored, or the sdkconfig.board is ignored or only the custom partition table option...? So far I found no clue why no OTA Image is created. Do you have any ideas what I could check? Thanks & best regards

[gampam2000]

I found that there are various sdkconfig files that are combined. I needed to remove duplicate entries of:

CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-8MiB-ota.csv"

and use them only once in the sdkconfig.board

This solved it for me.