geky
commented
3 years ago This is something I want to address in the future. There are a couple discussions floating around that I need to collect, but basically the underlying file data-structure (CTZ skip-lists) doesn't work well with random block changes. I'm currently of the opinion this data structure needs to change, but it will take some time to implement.
You're absolutely right that in theory you can COW the single block and avoid copying the other data, but the challenge is implementing that with O(1) RAM. Keep in mind, even traditional tree traversal needs O(log n) RAM.
My current thinking is to replace the CTZ skip-lists with some form of B-tree, either using additional metadata-pairs as non-leaf nodes or using variable-radix integers to track traversal over a strictly balanced tree.
But before jumping into this too far, I do want a cohesive set of benchmarks similar to the tests+scripts/test.py. The lack of performance measurements is sort of the reason for a lot of littlefs's current performance pitfalls. Aside from that, it's a long roadmap, but any performance measurements/double checking would be very welcome as these features get prototyped.
ullix
After experiencing multiple file corruptions with FFat on an ESP32 in the Arduino environment, I was happy when LittleFS became available to me. Initial testing using the append mode (open(file, "a")) showed LittleFS performing as good or even better than FFat. However, when I used overwriting (open(file, "rb+")) a very serious problem showed up. Detailed results are published on Github: https://github.com/lorol/LITTLEFS/issues/10.
The situation is as follows: when starting with a file which occupies 45% of the free flash space, and then seeking to position 0 (zero), and then writing e.g 100 bytes, LittleFS apparently has to copy the whole 45% to a different location, before completing the write. In my case this took 11 seconds to write 100 bytes, while appending 100 bytes to a file had always been done in under 0.22 seconds!
The situation is even worse when the file is only slightly larger, let's say 50.4%. Then overwriting data at the zero position requires that same free space of 50.4%, while obviously at most only 49.6% are available. The writing does take the time as if it were written (13 seconds) but actually nothing is written at all, and no error is reported.
I am wondering whether all this moving of data above the intended writing position is necessary? Couldn't it be done by moving just one block (4096 bytes), writing to it, and taking care of the remainder of the file by juggling the pointers to all those other blocks?
As it stands I have to go back to FFat, which does all this correctly, except that I now experience file corruption again. Wish I could use LittleFS!
I don't feel competent enough to work on file system code, but I'd be happy to support with testing. Thanks.