search

littlefs-project / littlefs

A little fail-safe filesystem designed for microcontrollers
BSD 3-Clause "New" or "Revised" License
4.92k stars 774 forks source link

Write time increasing with time/file size #860

Closed xgroleau closed 11 months ago

xgroleau commented 11 months ago

Currently I have a long lived file and we are writing data for a long duration (multiple Mb). Though it seems the write operation becomes longer and longer as we add more data to the file and eventually it starves the MCU causing other operations to timeout. It starts at less than 1ms to write the file content to more than 100ms after adding a lot of data.

Note that we are writing a lot of small chunk of data (5 bytes) but we are using a cache for each file and it seems the number of operations (read/write accesses) increases with the length of the file.

Is this a known fact that the write operations are proportional to the file size? We are using NAND flash but adapted the driver to offer a similar interface to NOR flash so it shouldn't be an issue. Is there anyway to increase the speed? Like periodically sync the file to the filesystem?

Here's our current setup:

LittleFS version: 2.5.1 (also tested with 2.7.0, same issue) NAND chip: w25n01gw READ_SIZE: 1 WRITE_SIZE: 512 BLOCK_SIZE: { 64 * 2048 } BLOCK_COUNT: 1000 BLOCK_CYCLES: 256 METADATA_MAX: 2048 CACHE_SIZE: 512 LOOKAHEAD_SIZE: 512

xgroleau commented 11 months ago

It seems buffering the writes quite dramatically (more than 10x). Not sure why since I am using a cache and I thought it should do it internally.

e107steved commented 11 months ago

Are you doing random writes (rather than simply appending)? If so, that's your problem. The nature of the file structures mean that data after the insertion point has to be rewritten. Search the "issues"; there are some far better explanations of this than I can give.

xgroleau commented 11 months ago

Hi, i'm doing sequential write. After some digging, I've found the info I was searching for in issue #783 so I'll close this issue. Buffering the write also helped quite a bit.