Too inefficient way to append file content, when i need to save file immediately

[muguou]

I use W25Q128 Flash. My LFS adopts the following configuration.

const struct lfs_config fsCfg =
{
    .read_size = 32,
    .prog_size = 32,
    .block_size = 4096,
    .block_count =10,       //address start at 0x800
    .lookahead_size = 64,
    .block_cycles = 8,
       .cache_size = 1024,
};

I have some data that needs to be stored in flash at any time when there is a power outage. So every time I use lfs to open a file (use open type writeOnly, append) and write to it, I immediately close the file (I also used sync testing, and the results were the same).

If I don't close file it or synchronize, then the power outage data will be lost at this time

The data is stored in a structure with a size of 64B, and the size is always 64 when writing. Because it immediately shuts down after each write, lfs saves the data back to flash after each write. In this case, when the file size is over 512 B, every time I save data, lfs erases a new sector. You know, I only wrote 64B of data, but every time lfs erases an entire sector of 4096B, there is a difference of 2 ^ 6=64 times. This is a bit too inefficient, and even less efficient than simply erasing the sectors in place.

Is it caused by my usage error.

My log is as follows。 The "i" mean times of write. "size" means the total number of bytes in the file now( befor write data and close file ). "fe" means erase sector .

- > 
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 0,size=0 (file size now)
- > w25WrAdd = 0x801040,size=64    //maybe creat file
- > w25WrAdd = 0x801080,size=96   //maybe write file =64; update file info = 32; total 96
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 1,size=64 
- > w25WrAdd = 0x8010E0,size=160   // maybe write file =64*2; update file info = 32;total 160. 
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 2,size=128 
- > w25WrAdd = 0x801180,size=224  // maybe write file =64*3; update file info = 32;total 224. Copying the entire file every time is too inefficient
- > 
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 3,size=192 
- > w25WrAdd = 0x801260,size=288
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 4,size=256 
- > w25WrAdd = 0x801380,size=352
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 5,size=320 
- > w25WrAdd = 0x8014E0,size=416
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 6,size=384 
- > w25WrAdd = 0x801680,size=480
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 7,size=448 
- > w25WrAdd = 0x801860,size=544
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 8,size=512 
- > fe:802          //when size is over 512 ,every time close file, erase sector. 
- > w25WrAdd = 0x802000,size=576
- > w25WrAdd = 0x801A80,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 9,size=576 
- > fe:803
- > w25WrAdd = 0x803000,size=640
- > w25WrAdd = 0x801AA0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 10,size=640 
- > fe:804
- > w25WrAdd = 0x804000,size=704
- > w25WrAdd = 0x801AC0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 11,size=704 
- > fe:805
- > w25WrAdd = 0x805000,size=768
- > w25WrAdd = 0x801AE0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 12,size=768 
- > fe:806
- > w25WrAdd = 0x806000,size=832
- > w25WrAdd = 0x801B00,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 13,size=832 
- > fe:807
- > w25WrAdd = 0x807000,size=896
- > w25WrAdd = 0x801B20,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 14,size=896 
- > fe:808
- > w25WrAdd = 0x808000,size=960
- > w25WrAdd = 0x801B40,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 15,size=960 
- > fe:809
- > w25WrAdd = 0x809000,size=1024
- > w25WrAdd = 0x801B60,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 16,size=1024 
- > fe:802
- > w25WrAdd = 0x802000,size=1024
- > w25WrAdd = 0x802400,size=64
- > w25WrAdd = 0x801B80,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 17,size=1088 
- > fe:803
- > w25WrAdd = 0x803000,size=1024
- > w25WrAdd = 0x803400,size=128
- > w25WrAdd = 0x801BA0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 18,size=1152 
- > fe:804
- > w25WrAdd = 0x804000,size=1024
- > w25WrAdd = 0x804400,size=192
- > w25WrAdd = 0x801BC0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 19,size=1216 
- > fe:805
- > w25WrAdd = 0x805000,size=1024
- > w25WrAdd = 0x805400,size=256
- > w25WrAdd = 0x801BE0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 20,size=1280 
- > fe:806
- > w25WrAdd = 0x806000,size=1024
- > w25WrAdd = 0x806400,size=320
- > w25WrAdd = 0x801C00,size=32
- > 
- > 
- > .............................Omit some conten
- > 
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 62,size=3968 
- > fe:806
- > w25WrAdd = 0x806000,size=1024
- > w25WrAdd = 0x806400,size=1024
- > w25WrAdd = 0x806800,size=1024
- > w25WrAdd = 0x806C00,size=960
- > w25WrAdd = 0x800180,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 63,size=4032 
- > fe:807
- > w25WrAdd = 0x807000,size=1024
- > w25WrAdd = 0x807400,size=1024
- > w25WrAdd = 0x807800,size=1024
- > w25WrAdd = 0x807C00,size=1024
- > w25WrAdd = 0x8001A0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 64,size=4096 
- > fe:808
- > w25WrAdd = 0x808000,size=96       //size over 4096 ,repeat the process
- > w25WrAdd = 0x8001C0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 65,size=4160 
- > fe:802
- > w25WrAdd = 0x802000,size=160
- > w25WrAdd = 0x8001E0,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 66,size=4224 
- > fe:803
- > w25WrAdd = 0x803000,size=224
- > w25WrAdd = 0x800200,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 67,size=4288 
- > fe:804
- > w25WrAdd = 0x804000,size=288
- > w25WrAdd = 0x800220,size=32
- > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>i= 68,size=4352 
- > fe:805
- > w25WrAdd = 0x805000,size=352         //size over 4096 ,repeat the process
- > w25WrAdd = 0x800240,size=32

---

Is it my usage mistake, or is it how lfs is designed.

[muguou]

Create a file for each data and name it with a number as the suffix.But this will make the entire logic more complex and create a large number of files, it is unknown whether this will significantly reduce the performance of the file system. Although I am more concerned about data validity and space utilization efficiency compared to performance, low file system performance may lead to watchdog restarts.

[muguou]

It has to be said that there is still room for optimization of the LFS system's strategy for large files. In fact, files without sync or close are not saved to flash. Even if 16KB of data has been written, in my example, it means 4block because it exceeds the cache and has been written to flash. However, if there is no sync or close, the data cannot be successfully read. So if you want to store and read data that can be accessed at any time in flash, you must perform a sync or close operation after each write.

However, LFS's write strategy for large files is frustrating. Even if I add 1 Byte of content to this outline file, LFS will erase a new sector and then copy the contents of this 1 Byte and the previous content.

For example, if a file now has 4096+512Byte, the entire file takes up 2Block,

1 block 4096 data; 2 block 512 data 32 index. Obviously, there are still 3552B unused in block 2.

If I add 1 byte of data at this time, lfs will erase a new block to store the updated data.

Delete 2 block, move to 3 block

At this time, 1 block 4096 data; 3Block 513data 32 index.

This clearly wastes a large amount of the remaining space of the 2block, which is nearly 7 times the space used. It is difficult to imagine the rationality of this approach.

It is worth noting that if 1B more data is added, lfs will also repeat the above operation!

This waste of space confuses me.

[muguou]

Additionally, I noticed that updating the inline file is more reasonable and utilizes space more effectively. Can several inline files be used to maintain a large outline file without changing the source code? In short, the current outline file scheme is not a reasonable one

[geky]

Hi @muguou, thanks for creating an issue, the feedback and ideas are useful.

This is a long-term issue I am currently working on, along with a number of other file data-structure changes, though it will take some time to implement and test.

I think I've more or less arrived at the same conclusion you have, that is, inlined data can be utilized better. Though any inlined data has an inherent 4x storage overhead, so there is a bit of a tradeoff.

The current plan, susceptible to change:

1. Detach the RAM requirement for inline data. This would allow inline data to get quite large without an unreasonable RAM requirement.
2. Allow a fully inlined tree of inlined data slices. This is the least clear change, since it risks adding quite a bit of complexity, and complexity brings code cost. But it may be necessary to avoid $O(n^2)$ write costs when writing large inlined files.
3. Allow non-inlined files to have inlined data that sort of overlays the non-inlined data. This would allow sync calls to save the cache as inlined data without needing to write potentially unaligned data to raw blocks. Which is the underlying cause of the block rewrites.

In theory, these combined should eliminate the issues with syncing files. As long as inline data is larger than the program size, writes can go into the file's inlined data until program aligned. So we should be able to avoid block rewrites even when repeatedly syncing.

Create a file for each data and name it with a number as the suffix.But this will make the entire logic more complex and create a large number of files, it is unknown whether this will significantly reduce the performance of the file system.

Currently file lookup is a $O(n)$ operation, so this is a valid concern.

Though it may still be more efficient thank one large file with the current implementation, depending on cost of rewriting the last block in a file.

[Olstyle]

I am a bit confused reading this issue, but can see the same behavior on my system. In most issues regarding logging etc. (e.g .https://github.com/littlefs-project/littlefs/issues/104) it is advised to use append as it should be the most efficient way. Now I see that every append is one block erase, which isn't effective at all. Why then give the original advise?

[geky]

@Olstyle, that's easy to answer. I'm not perfect and littlefs (any filesystem really) is a complex system.

The intention is for append to be the most efficient solution, and after the planned changes in https://github.com/littlefs-project/littlefs/issues/862#issuecomment-1681317398 (though this will take some time), it should return to being the most efficient solution for logging.

I'd also be hesitant to suggest the above many-files solution as always better than append. It's a workaround for current design issues that may have unpredicted side-effects. For 1. lookup cost in directories is currently $O(n)$, so many-files may not be great for read-heavy applications. 2. inlined files use more storage than uninlined files, at minimum 4x, and more if you consider each file has a filename + other metadata attached.

Append is also still the most efficient solution if you don't need to sync, though I realize that's not a good idea for logging. Streaming sensor data however...

[kyrreaa]

How is a changed block handled now? It seems a new block is allocated and then all old data is copied with new data applied over the affected bytes. Then all later blocks are copied to new blocks to allow references to be updated? Or, does it make a copy of the affected block, then write it back to same block number to keep all later blocks valid? The latter does not wear level but creates less total wear and would be faster?

[geky]

Hi @kyrreaa, the answer is your first description. littlefs creates a copy of any blocks being modified, and then updates any dependent blocks so references are up to date.

Modifying the affected block is not an acceptable strategy here. Wear-leveling aside, it breaks power-loss resilience, as a power-loss could leave us in a state where the block is only partially written and metadata is not up to date.

You're right it would be faster though! Which is why littlefs will probably always be less-performant than non-power-loss-resilient filesystems.

[ericmbf]

Hi guys, I'm doing a lot of tests to check the LFS performance. I'm trying to log events(small files, 32 bytes for a while) as quickly as possible. I was trying this approach below.

Create a file for each data and name it with a number as the suffix

After frustration tests trying to create 200 logs files with the suffix and delete the oldest files( The performance decreased linearly), I decided to overwrite the oldest files and maintain the same files names log1.txt ... log200.txt (circular buffer index to know the oldest and newest). I figured out a constant performance in this approach, no more increasing over the time and random open/close times. I did some tests creating folders and puts the files inside. I could see a better performance ( Folder 1 -> log1, ..., log20, Folder 2 -> log21, ..., log40 ...) than 200 files in the root filesystem.

Below is the difference between using folders(10 folders to store 200logs) and putting the same files in the root directory. In this case, I could see a huge difference in the metadata compact spikes.

I also set the parameter:

metadata_max = 512

To reduce the spikes and have an acceptable performance for my application.

I think this confirm what @geky said:

I'd also be hesitant to suggest the above many-files solution as always better than append.

I think for this LFS algorithm logic, as less files do you have, better the performance. I didn't try big files yet. But for small one, looks better breaks it in folders when its possible.

I hope this can help someone else that are having performances issues with small log files.