Flash Translation Layer

[deshipu]

I'm creating this issue to collect information about possible solution for a flash translation layer for CircuitPython. While the flash memories CircuitPython boards normally use have a reasonably good lifetime, the FAT filesystem is not very friendly for flash. Adding the complexity of a wear-leveling layer would normally be an overkill, but I have recently discovered that there are some ready to use libraries for doing this, and some of them are optimized for low-resource systems. It might make sense to use one of them for the flash file system at some point.

[deshipu]

One such library is https://github.com/dlbeer/dhara It's BSD-licensed, and it's optimized for low-resource systems.

[C47D]

Here's another FTL for NOR memories used on the Hackaday Supercon 2019 badge (it doesn't include a license file): tjftl

[tannewt]

@C47D That looks perfect! @deshipu thank you for creating this issue.

[tannewt]

ESP IDF has this as well: https://github.com/espressif/esp-idf/blob/master/examples/storage/wear_levelling/README.md

[EternityForest]

Would it be possible to just use LittleFS for everything? FAT isn't power failure tolerant, which would be a nice feature to have .

The PC USB side could use a virtual network device and a web file manager, or MTP, or a dedicated app, or an emulated fake FAT32 interface, as has been done on some boards.

Or, could there be some generic "Advanced" builds for some of the ports than just used LittleFS and expected the user to figure out file transfer themselves?

[dhalbert]

Early on, we considered MTP. But there is no MTP support for macOS, sadly.

Is your concern about FAT that you are doing local writes in CircuitPython code to the filesystem, or that writes from the host might be interrupted?

CircuitPython has chosen FAT because it makes the workflow easy: no extra programs are required on the host computer. We prevent some problems by making the filesystem readonly to the CircuiPython code but not to the host, or vice versa. When MicroPython used FAT more, they did not do this. MicroPython has been going in the direction of suggesting the use or mpremote or Thonny, and not exposing a FAT filesystem via USB.

[EternityForest]

Ah, I thought there might be a catch! Apple compatibility issues always seem to cause trouble!

My big concern is that any application that periodically writes to flash will wear much faster than it needs to, and not have any power failure tolerance.

It seems like Dhara's checkpointing features would at least partially fix the power fail issues though, as long as there's not too many writes in between closing or syncing a file.

I assume FatFS probably syncs at smart times as one would expect, not in the middle of metadata operations.

Is Dhara or similar a possibility? Seems like having it done before any rumored v10 partition changes would be good.

[dhalbert]

In practice, we have not had reports of flash wearing out. People who are doing a lot of logging and the like tend to use SD cards.

I had not heard of dhara before, and found it here: https://github.com/dlbeer/dhara. Thanks for the pointer. I don't know if it is still being maintained.

I assume FatFS probably syncs at smart times as one would expect, not in the middle of metadata operations.

I'm not sure what you mean here. The writes to flash are cached in RAM and are done periodically. FatFS itself does not do the writes. There's code in supervisor/shared that handles that.

[dhalbert]

Ah, I thought there might be a catch! Apple compatibility issues always seem to cause trouble!

They didn't want MTP because it allowed support of competitors to iPod. I think it's a dead issue now but they have no incentive to implement MTP now because it's not used very much, but that's partly a chicken-and-egg problem. We'd love to be able use a higher-level transactional filesystem on the host side.

[EternityForest]

I'm not sure what you mean here. The writes to flash are cached in RAM and are done periodically. FatFS itself does not do the writes. There's code in supervisor/shared that handles that.

I just found Dhara in a comment earlier up in the thread, but it seems to have some limited atomic transaction support.

So if you create a new checkpoint every time FatFS issues a sync ioctl, and you don't do too much in between syncs(It automatically makes its own checkpoints periodically), then all of the checkpoints should be at times where the filesystem is valid, and power failure shouldn't be an issue unless you do a ton of writes at once, or there's a sync call somewhere at an invalid time.

[dhalbert]

The problems we see with corrupted filesystems are mostly due to:

• User presses reset when the host is writing or there's a write happening on the board side.
• User unplugs (same issues)
• Host delays writes for a lengthy period of time.
  • This was a big problem with Windows a long time ago, and supposedly was fixed in Windows 10 not too long ago (a year or two or three?). In particular, FAT12 had much longer write delays than larger FAT formats, which may have been left over from the days of floppies. See #111.
  • Linux has modest delays, but waiting a few seconds or typing sync or the equivalent fixes the issue.
  • macOS used to write immediately, and had the fewest problems. But starting in Sonoma, they started reimplementing pieces of FAT support in userspace, and introduced very long delays, including between writing data and metadata. At least in the other filesystems the writes happened all at once if there were long delays. They partly fixed this, but at the expense of 40x slower writes for filesystem 1GB or less. See #8449 and #8918.
  • Some editors do not do a flush() or the equivalent after writing a file. See https://learn.adafruit.com/welcome-to-circuitpython/recommended-editors.

These problems are all far more prevalent than power glitches, etc. The basic FatFS writing and caching seems fine, and when it isn't, we consider it a bug and have fixed it.

[bablokb]

Linux has modest delays, but waiting a few seconds or typing sync or the equivalent fixes the issue.

@dhalbert : the simplest solution is to mount the CIRCUITPY-drive with mount -o sync .... There will be no OS-level buffering anymore and once the file is closed it is on the drive. This also fixes problems with editors not doing the flush().

[tannewt]

My big concern is that any application that periodically writes to flash will wear much faster than it needs to, and not have any power failure tolerance.

What devices are you interested in using? We could switch all non-native USB devices over to littlefs on a major version boundary. Only USB MSC exposes the block level to an external computer. Without it, we could switch to littlefs.