Question: Trying to not wear the memory...

[ElVasquito]

Hello, I have a question about implementing LittleFS in my ESP8266 project. I am calling a Receive() function which reads various wireless PIR sensors. I want to log the information on the FS and append data to 5 files, but want to prevent closing or syncing the files every time the function gets called, to try to prevent wear on the memory. Packets are between 6 and 50 bytes in size. My idea is to have a byte buffer, and syncing once it fills. Is 256 bytes OK to write without wearing the memory each time? Or should I use more? Or can LittleFS do it automatically when cache size is hit? I have 4mb of flash, with 2MB LittleFS partition. Block size 8192, page size 256. I did a default LittleFS.format() and I'm not using any custom partition parameters. Thanks for info!

[geky]

Hi @ElVasquito, sorry about late response.

The cache in LittleFS should already do the buffering for data before a sync for you. If you set cache_size = 256, you can write 256 bytes to a file before any of the data is written to disk. This buffer is per-file, so multiple files won't cause data to get written to disk early.

Judging from your comment, are you using Mbed OS? You may need to switch to LittleFS v2 (here) to get the cache_size config option.

Is 256 bytes OK to write without wearing the memory each time?

The buffering won't save wear on the data, but it does reduce the amount metadata that gets written. A sync will write roughly 12 bytes of metadata to indicate the new position of the file. On top of this is periodic garbage collection, which will write ~1/2 of your block size every time a block is full. Amortized this is an extra 6 bytes per metadata entry.

How much wear is ok can be a hard question. If you are concerned you can actually calculate how long your flash is expected to last.

The data sheet for flash parts will say how many erase cycles you can expect. It looks like the ESP8266's external flash is different based on which board you are using, but if it's a SPI NOR flash chip, we can expect something on the order of 100K erase cycles.

So, if you sync once every second, on a flash part with 2MiB, where each block has 100K erase cycles. You can expect, optimistically, ~200G of update bytes. So 18 bytes every seconds gives you, very optimistically, ~350 years before the flash dies from syncs alone. I would divide that by 2 to be safe, since not all blocks last the same amount of time, and there are other bookkeeping operations littlefs may do. So maybe ~125 years of only sync calls.

Hopefully that helps a bit getting you started.

[ElVasquito]

I already implemented a 256 byte buffering scheme which seems to work (hehe). But the cache option looks great to consider. I am using Arduino with the ESP8266 library, which includes LFS v2.

periodic garbage collection, which will write ~1/2 of your block size every time a block is full

So now I understand why having two files of 33536 and 4419 bytes respectively, for "used space" I get 81920 bytes. I suppose that this Garbage Collection also produces wear, apart from taking some huge space... :) Thanks for the explanation!

[geky]

So now I understand why having two files of 33536 and 4419 bytes respectively, for "used space" I get 81920 bytes. I suppose that this Garbage Collection also produces wear, apart from taking some huge space... :)

Ah, yes, the garbage collector introduces wear. To fight this, littlefs has two-layers of wear-leveling: super-block-level and sub-block-level. Large files get their own blocks which only participate in the super-block-wear-leveling, but their metadata gets stored in a "metadata pair", which has its own sub-block-wear-leveling together with other metadata in a directory.

It gets a bit complicated, in an effort to reduce the wear as much as possible : )

There's more info about this here, don't know if it might be helpful: https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md#wear-leveling

[geky]

Oh, but the downside of putting the files in their own blocks is they are rounded up to the nearest block.

So file1 = 33536 gets rounded to 40960 bytes, file2 = 4419 gets rounded to 8192 bytes, and, as its name hints, the root directory's "metadata pair" takes two blocks = 16384 bytes. So a total of 65536 bytes.

Ok, so 65536 bytes != 81920 bytes. I think this can happen if you have both files open for writing when you call lfs_fs_size, since littlefs will count the temporarily allocated blocks for the two files as "in use".

[ElVasquito]

Hmmm, nope... Not at all. File1 is the one that gets written once the 256-byte buffer fills. It happens between 1-3 hours. I open the file, write the buffer, and close it right away. File2 is a config file that doesn't change often. I read it on startup and close it. So neither file is open at the moment I call LittleFS.info() to get the usedBytes... Right now for example on the FS I have: 1:2019-December.dat 5632 2:2019-November.dat 105728 3:config 1611 LittleFS.info says: Total: 2072576 Used: 155648

The ESP8266 wrapper function for LittleFS.info():

    bool info(FSInfo& info) override {
        if (!_mounted) {
            return false;
        }
        info.maxOpenFiles = _maxOpenFds;
        info.blockSize = _blockSize;
        info.pageSize = _pageSize;
        info.maxOpenFiles = _maxOpenFds;
        info.maxPathLength = LFS_NAME_MAX;
        info.totalBytes = _size;
        info.usedBytes = _getUsedBlocks() * _blockSize;
        return true;
    }

    int _getUsedBlocks() {
        if (!_mounted) {
            return 0;
        }
        return lfs_fs_size(&_lfs);
    }

It is here: https://github.com/esp8266/Arduino/tree/master/libraries/LittleFS Maybe this implementation has something to do with the size difference you mention?