search

adafruit / tinyuf2

UF2 bootloader based on TinyUSB for embedded devices such as ESP32S2, STM32F4 and iMX RT10xx
MIT License
298 stars 165 forks source link

Like CURRENT.UF2, could there be EVERYTHING.UF2 w/ entire flash filesystem? (code.py,libs,etc) #117

Open todbot opened 3 years ago

todbot commented 3 years ago

A Discord discussion led me to want the ability to have something like CURRENT.UF2, but contained both the current version of CircuitPython and the contents of the CP filesystem (code.py, lib, etc)

Would it even be possible for this hypothetical EVERYTHING.UF2 virtual file to be created? I know this is outside of the purview of the bootloader, but if the bootloader takes an entire snapshot of all known flashes and restores them, would it work?

hathach commented 3 years ago

it depends on the port, can you tell which board you are using ?

todbot commented 3 years ago

I was hoping it could be platform-independent. I use almost all the CircuitPython platforms pretty regularly: ESP32-S2, SAMD, and RP2040. (I understand RP2040 is ROM-based UF2)

hathach commented 3 years ago

Platform-independent still need port specific implementation to pull this off.

T94T commented 3 years ago

We are also interested in exact the same feature. Mainly on ESP32S2.

henrygab commented 2 years ago

For chips that use a partition, perhaps it would be possible to update to expose each of the partitions as UF2 binaries?

This would require:

  1. ability for boards to indicate, at initialization time (uf2_init), at least: a. the number of partitions to expose b. the start and end address for each exposed partition c. optionally, an 8.3 fat-compliant filename for each exposed partition
  2. update ghostfat to: a. use embedded-safe C++11 b. rework major portions to dynamically calculate (at uf2_init) values, rather than current #define compile-time constants (but: check RAM increases OK?) c. rework how files are handled (current code presumes exactly one UF2 file exposed)
  3. define "graceful" failure modes ... esp. for cases that are currently prevented by compile-time assertions, but will move to dynamic validation

Embedded-safe C++

Sometimes called "C++ as a better C", it restricts what features of C++ are used. For example, absolutely prohibited features include:

  • exceptions
  • run-time type informatino (RTTI)
  • dynamic allocation (e.g., most of STL)

Usable features that provide value include:

  • constexpr functions ... safer macros!
  • scoped enums
  • namespaces
  • libraries that enable zero-runtime-overhead type-safety (imagine! FAT cluster numbers no longer causing off-by-two errors!)

This would definitely take a while to implement, because the current ghostfat code has many compile-time constants and behaviors that ensured via static_assert. Changing those would increase testing, and the need to code defensively (runtime-checks, instead of compile-time ... and how to fail gracefully?).

hathach commented 2 years ago

@henrygab thank you for your informative input, though, I think bootloader should just focus on updating the firmware anything that relates to firmware usages such as its file system should be done by firmware itself. For embedded C++, I don’t think we need to migrate to it since not all MCUs have C++ compiler for it and we actually don’t have to do complicated thing within bootloader. I agree with defensive coding style, but it is best to write as little code as possible.

henrygab commented 2 years ago

Ok, you're the boss! 👍

tannewt commented 2 years ago

I believe that SAMD's CURRENT.UF2 already includes the filesystem. nRF may as well. It is definitely a handy feature.

hathach commented 2 years ago

@tannewt SAMD & nRF52 CURRENT.UF2 includes all internal flash contents therefore also include internal file system e.g nvm if available but not the file system on external flash. The S2/S3 has different partitions for firmware, nvm, file system and they may not be continuous, You probably understand this better than me.

For S2/S3, currently tinyuf2 limit itself within the firmware(ota) partition to prevent overwriting to other partitions. Therefore the address in uf2 is starting from zero 0x00 as start of partition. However we could enhance this behavior and dump everything into CURRENT.UF2 (after partition table and 2nd stage bootloader), and when writing back we simply write over everything. To prevent bricking incompatible partition table and/or different flash size e.g 4 MB vs 8 MB we could use vendor + product ID for the family ID to make sure it can only be written to the same board. Let me know if you have any other suggestion.

Ok, you're the boss! +1

Nah, I wouldn't want bootloader to know too much of app knowledge and just want to write as less code as possible. Putting them all into CURRENT.UF2 is better way to go.

NOTE: currently max size is around 32MB with 1 cluster = 1 sector config, which limit the flash chip to less than 16MB. To support 16MB and more, we need to enhance ghostfat to run with 1 cluster = n sectors (n to be decided later).

tannewt commented 2 years ago

Very good points @hathach. It'd probably be best to chat with the Microsoft UF2 folks about how we designate UF2s for different uses (aka partitions.) We wouldn't want to copy off the nvm partition data and then copy it over an ota section.

henrygab commented 2 years ago

FYI:

ESP32 appears to be hard-coded to only allow writes to the ota1 partition.

Moreover, the offset stored in existing .uf2 files is a relative offset for that ota partition.

Ghostfat has no concept of which file is being written to, and IO can be re-ordered by the time the device sees it (e.g., no guarantee that sector 0 will be seen as written by device before sector 10). Therefore, you cannot simply insert a "switch to partition X" (or any stateful) command into the head of the .uf2 file, as it may be applied before the device sees writes that should have been affected by that stateful command.

In combination, these factors make it challenging to support writing more than a single target partition in ghostfat. However, if you simply want to expose additional .uf2 files to allowing reading of other partitions (such as a file system), that would be (still non-trivial, per my earlier answer), but within the realm of possibility. But without the ability to write it back, what would be the value?

expand for links to relevant portions of code

The target address is embedded in each sector of the .UF2 file. See targetAddr field in uf2.h: https://github.com/adafruit/tinyuf2/blob/9104f7c114086d52bb1bbd03bd5071333d91ed0a/src/uf2.h#L86-L95

Ghostfat provides the buffer provided by the .UF2 sector to the underlying board: https://github.com/adafruit/tinyuf2/blob/9104f7c114086d52bb1bbd03bd5071333d91ed0a/src/ghostfat.c#L498

As an example, the Espressif port caches stuff: https://github.com/adafruit/tinyuf2/blob/06a65c824d00112ab5d5f8b55cfa5b0466d80ffa/ports/espressif/boards/board_flash.c#L109-L122

But... the board_flash_read() and board_flash_write() functions are hard-coded to only allow writing to the ota0 partition: https://github.com/adafruit/tinyuf2/blob/06a65c824d00112ab5d5f8b55cfa5b0466d80ffa/ports/espressif/boards/board_flash.c#L72-L75

hathach commented 2 years ago

FYI:

ESP32 appears to be hard-coded to only allow writes to the ota1 partition.

Moreover, the offset stored in existing .uf2 files is a relative offset for that ota partition.

Yup, currently tinyuf2 for esp32s2/s3 is writing as offset to ota0 to prevent overflow to other partition.

In combination, these factors make it challenging to support writing more than a single target partition in ghostfat. However, if you simply want to expose additional .uf2 files to allowing reading of other partitions (such as a file system), that would be (still non-trivial, per my earlier answer), but within the realm of possibility. But without the ability to write it back, what would be the value?

I think the most wanted featured is not exposing additional partition as uf2. User want to take the snapshot of the current working state of application(firmware + data) and then clone/deploy/duplicate it to another board. Therefore I think packing all the flash together within a single all-in-one UF2 make more sense. However, since the current code use address as offset to 1st partition (start address = 0), I would suggest to use vendor+board id as family for the all-in-one uf2 in addition to current offset scheme.

todbot commented 4 months ago

Coming back to this after 3 years: having a single UF2 to install would make loading up all the code on the MEMENTO board a lot easier.