Open Motirn opened 6 months ago
@geky Any thoughts on this ^^ plz
Hi @Motirn,
Correct me if I have any of your values wrong:
Am I right you are calling sync every write?
I think this is the file padding issue, seen also in https://github.com/littlefs-project/littlefs/issues/862. Long story short littlefs currently does not track erased state/alignment in files, so appending to files always requires rewriting (with erase) the last block in the file. Since your blocks contain the whole file, this means littlefs is rewriting the whole file.
You shouldn't see this behavior if you don't call lfs_file_sync
. But I realize that is a non-option for a number of use cases.
The real fix (tracking erased-state) is in the works, but due to other things it is getting rolled up with bigger changes to the file data structure. Which unfortunately means it will take some time before it's usable... On the plus side the erased-state checksums necessary to make this work are passing testing on this experimental branch... Though I'm not sure that's consolation...
Hi @geky Thank you for your thoughts on this. Yes, the values are correct, Files are compressed audio files,and expect it to be around 120-150KB in size. Is the write size of 4KB optimal ? If not, Can you please suggest some values that might improve write efficiency? At the moment the sync() do not do anything on my flash driver layer, and removing it has vastly improved the speed.
At the moment the sync() do not do anything on my flash driver layer, and removing it has vastly improved the speed.
That's good to hear. Unfortunately that's probably the best workaround for now...
Is the write size of 4KB optimal ?
You could perhaps decrease it to the minimum 2KiB, generally smaller values for read_size
/prog_size
are better, since it allows for smaller commits/less padding. But I don't know if it will really have that much of an impact on performance.
At the end of the day, the best thing to do and benchmark and tweak things to see if they improve. It's hard to always know the best configuration given the number of knobs.
define PAGE_DATA_SIZE 2048
define FILE_CACHE_BUF_SIZE (4* PAGE_DATA_SIZE)
uint8_t lfs_read_buf[4 PAGE_DATA_SIZE];
uint8_t lfs_prog_buf[4 PAGE_DATA_SIZE];
uint8_t attribute((aligned(8)))lfs_lookahead_buf[64]; struct lfs_config lfs_cfg = { .context = (void)"spi_nand_flash", .read = read_flash_block, .prog = prog_flash_block, .erase = erase_flash_block, .sync = sync_fs, .read_size = PAGE_DATA_SIZE, .prog_size = PAGE_DATA_SIZE, .block_size = 64PAGE_DATA_SIZE, .block_count = 2048, .block_cycles = 500, .cache_size = FILE_CACHE_BUF_SIZE, .lookahead_size = 64, .read_buffer = lfs_read_buf, .prog_buffer = lfs_prog_buf, .lookahead_buffer = lfs_lookahead_buf, .metadata_max = 8192, };
----A loop writing to a file runs until 'filesize' becomes zero, 120KB written to file, Each write is 4KB---------
[02:10:42.845,520] littleFS: lfs_file_opencfg(0x2000dca4, 0x2000dc4c, "44f", 903, 0x2000db40 {.buffer=0x200146cc, .attrs=(nil), .attr_count=0})
[02:10:42.936,431] spi_flash: LFS:sync_fs: ----
[02:10:42.980,560] littleFS: lfs_file_opencfg -> 0
[02:10:42.996,154] littleFS: lfs_file_size(0x2000dca4, 0x2000dc4c)
[02:10:43.011,779] littleFS: lfs_file_size -> 0
[02:10:43.022,277] spi_flash: FlashFSOpenFile:44f - size is 0 bytes
[02:10:43.037,902] littleFS: lfs_fs_size(0x2000dca4)
[02:10:48.227,172] littleFS: lfs_fs_size -> 278
[02:10:48.237,640] spi_flash: <<< Flash FS occupies 278 blocks >>>
[02:10:48.253,265] : ==== test loop ========= 1
[02:10:48.268,829] : ==== test loop =========2, filesize:122880
[02:10:48.284,454] littleFS: lfs_file_seek(0x2000dca4, 0x2000dc4c, 0, 2)
[02:10:48.300,140] littleFS: lfs_file_seek -> 0
[02:10:48.310,607] littleFS: lfs_filewrite(0x2000dca4, 0x2000dc4c, 0x200136cc, 4096)
[02:10:48.331,787] spiflash: FlashBlockErase - Erase block 0x2A7(679) on plane #1
[02:10:48.355,377] littleFS: lfs_file_write -> 4096
[02:10:48.370,971] spi_flash: FlashFSWriteToFile - Success writing file 44f, written 4096 bytes
[02:10:48.391,723] littleFS: lfs_file_size(0x2000dca4, 0x2000dc4c)
[02:10:48.407,348] littleFS: lfs_file_size -> 4096
[02:10:48.422,943] spi_flash: FlashFSWriteToFile - size:4096
[02:10:48.438,537] littleFS: lfs_file_sync(0x2000dca4, 0x2000dc4c)
[02:10:48.516,052] spi_flash: LFS:sync_fs: ----
[02:10:48.568,664] littleFS: lfs_file_sync -> 0
[02:10:48.579,162] i2s_audio: ==== test loop ========= 3
[02:10:48.594,726] littleFS: lfs_file_seek(0x2000dca4, 0x2000dc4c, 0, 2)
[02:10:48.610,412] littleFS: lfs_file_seek -> 4096
[02:10:48.625,976] littleFS: lfs_filewrite(0x2000dca4, 0x2000dc4c, 0x200136cc, 4096)
[02:10:48.647,949] spiflash: FlashBlockErase - Erase block 0x2A8(680) on plane #1
[02:10:48.785,400] littleFS: lfs_file_write -> 4096
[02:10:48.801,025] spi_flash: FlashFSWriteToFile - Success writing file 44f, written 4096 bytes
[02:10:48.821,746] littleFS: lfs_file_size(0x2000dca4, 0x2000dc4c)
[02:10:48.837,371] littleFS: lfs_file_size -> 8192
[02:10:48.852,966] spi_flash: FlashFSWriteToFile - size:8192
[02:10:48.868,560] littleFS: lfs_filesync(0x2000dca4, 0x2000dc4c)
[02:11:00.290,740] spiflash: FlashBlockErase - Erase block 0x249(585) on plane #0
[02:11:11.802,154] spi_flash: LFS:sync_fs: ----
[02:11:11.838,775] littleFS: lfs_file_sync -> 0
[02:11:11.849,243] : ==== test loop ========= 3.5
[02:11:11.864,837] : ==== test loop =========2, filesize:114688
[02:11:11.880,462] littleFS: lfs_file_seek(0x2000dca4, 0x2000dc4c, 0, 2)
[02:11:11.896,148] littleFS: lfs_file_seek -> 8192
[02:11:11.911,743] littleFS: lfs_filewrite(0x2000dca4, 0x2000dc4c, 0x200136cc, 4096)
[02:11:11.933,746] spiflash: FlashBlockErase - Erase block 0x2A9(681) on plane #1
[02:11:12.106,262] littleFS: lfs_file_write -> 4096
[02:11:12.121,917] spi_flash: FlashFSWriteToFile - Success writing file 44f, written 4096 bytes
[02:11:12.142,669] littleFS: lfs_file_size(0x2000dca4, 0x2000dc4c)
[02:11:12.158,325] littleFS: lfs_file_size -> 12288
[02:11:12.173,919] spi_flash: FlashFSWriteToFile - size:12288
[02:11:12.189,514] littleFS: lfs_file_sync(0x2000dca4, 0x2000dc4c)
[02:11:12.267,089] spi_flash: LFS:sync_fs: ----
[02:11:12.310,485] littleFS: lfs_file_sync -> 0
[02:11:12.320,983] : ==== test loop ========= 3
[02:11:12.336,547] littleFS: lfs_file_seek(0x2000dca4, 0x2000dc4c, 0, 2)
[02:11:12.352,233] littleFS: lfs_file_seek -> 12288
[02:11:12.367,828] littleFS: lfs_filewrite(0x2000dca4, 0x2000dc4c, 0x200136cc, 4096)
[02:11:12.389,831] spiflash: FlashBlockErase - Erase block 0x2AA(682) on plane #1
[02:11:12.677,581] littleFS: lfs_file_write -> 4096
[02:11:12.693,206] spi_flash: FlashFSWriteToFile - Success writing file 44f, written 4096 bytes
[02:11:12.713,958] littleFS: lfs_file_size(0x2000dca4, 0x2000dc4c)
[02:11:12.729,614] littleFS: lfs_file_size -> 16384
[02:11:12.745,208] spi_flash: FlashFSWriteToFile - size:16384
[02:11:12.760,803] littleFS: lfs_file_sync(0x2000dca4, 0x2000dc4c)
[02:11:12.797,760] spi_flash: LFS:sync_fs: ----
[02:11:12.850,402] littleFS: lfs_file_sync -> 0
[02:11:12.860,900] : ==== test loop ========= 3.5
[02:11:12.876,464] : ==== test loop =========2, filesize:106496
[02:11:12.892,089] littleFS: lfs_file_seek(0x2000dca4, 0x2000dc4c, 0, 2)
[02:11:12.907,775] littleFS: lfs_file_seek -> 16384
[02:11:12.923,370] littleFS: lfs_filewrite(0x2000dca4, 0x2000dc4c, 0x200136cc, 4096)
[02:11:12.945,343] spiflash: FlashBlockErase - Erase block 0x2AB(683) on plane #1