Doubt: how to provide statically allocated buffers to littleFs ?

[baldhead69]

Hi,

I was trying to open a file in littlefs, but a error, no_mem error appear, because i setted my compiler to 0 heap size.

How to provide statically allocated buffers to littleFs ?

Where i can put these buffers ?

File: lfs_util.h

// Allocate memory, only used if buffers are not provided to littlefs
// Note, memory must be 64-bit aligned
static inline void *lfs_malloc(size_t size) {
#ifndef LFS_NO_MALLOC
    return malloc(size);
#else
    (void)size;
    return NULL;
#endif
}

Notes: PIC32MZ MPLAB MCC ( Microchip Code Configurator ).

Thank's.

[e107steved]

a) You can assign static buffers through .read_buffer and .prog_buffer in the lfs_config structure. Not sure, but logically this method limits you to one open file at a time. b) You can open a file using lfs_file_opencfg() (see lfs.h) which allows you to pass a static buffer specifically for that file.

[baldhead69]

Hi @e107steved

is there any rule on how to specify the size of this buffer ?

Thank's.

[baldhead69]

The strange thing is that it seems to me that the MCC has configured these buffers. There must be some more configuration missing, some #define, maybe #define LFS_NO_MALLOC

File: sys_fs_littlefs_interface.c

uint8_t ReadBuf[LFS_CACHE_SIZE] = {0};
uint8_t ProgBuf[LFS_CACHE_SIZE] = {0};
uint8_t LookaheadBuf[LFS_LOOKAHEAD_SIZE] = {0};

BLOCK_DEV bd;
static const struct lfs_config cfg = {
        .context        = &bd,
        .read           = lfs_bdio_read,
        .prog           = lfs_bdio_prog,
        .erase          = lfs_bdio_erase,
        .sync           = lfs_bdio_sync,
        .read_size      = LFS_READ_SIZE,
        .prog_size      = LFS_PROG_SIZE,
        .block_size     = LFS_BLOCK_SIZE,
        .block_count    = LFS_BLOCK_COUNT,
        .block_cycles   = LFS_BLOCK_CYCLES,
        .cache_size     = LFS_CACHE_SIZE,
        .lookahead_size = LFS_LOOKAHEAD_SIZE,
        .read_buffer    = ReadBuf,
        .prog_buffer    = ProgBuf,
        .lookahead_buffer = LookaheadBuf,
    };

[geky]

Hi @baldhead69, the buffers all need to match the configured cache_size. It looks like you figured that out.

There are two places littlefs needs buffers provided:

1. In lfs_mount, in struct lfs_config for the "read_buffer", "prog_buffer", and "lookahead_buffer":

   Note that the lookahead_buffer needs to be 32-bit aligned and a multiple of 8.

   uint8_t read_buffer[LFS_CACHE_SIZE];
   uint8_t prog_buffer[LFS_CACHE_SIZE];
   // note lookahead buffer needs to be 32-bit aligned and a multiple of 8
   uint32_t lookahead_buffer[LFS_LOOKAHEAD_SIZE / 4];
   
   static const struct lfs_config cfg = {
      .cache_size = LFS_CACHE_SIZE,
      .read_buffer = read_buffer,
      .prog_buffer = prog_buffer,
      .lookahead_buffer = lookahead_buffer,
      .lookahead_size = LFS_LOOKAHEAD_SIZE,
      ...
   }
   
   int err = lfs_mount(&lfs, &cfg);

2. In lfs_file_opencfg, in struct lfs_file_config, each file needs a "file_buffer":

   uint8_t file_buffer[LFS_CACHE_SIZE];
   
   static const struct lfs_file_config file_cfg = {
      .file_buffer = read_buffer,
      ...
   }
   
   int err = lfs_file_opencfg(&lfs, &file, "filename", flags, &file_cfg);

   This prevents us from needing to know how many files will be opened beforehand, but is a bit unintuitive.

EDIT: Fixed the incorrect omission of the lookahead_buffer

[CalebDueck]

I've tried following the above steps but I'm getting a LFS_ERR_NOMEM error when I try to initialize the program with the heap set to zero. It is able to run if the heap is equal to the size of read_buffer + prog_buffer Should the above steps completely eliminate the need for heap memory? Or are the buffers stored on the heap even when statically allocated?

[geky]

Curious, yes the above steps should eliminate any memory allocations.

Are you sure there are no calls to lfs_file_open without a static buffer? A stack trace with an assert in malloc + GDB should tell you where the allocation comes from pretty quickly.

Are you able to share the code that returns LFS_ERR_NOMEM in some form?

[CalebDueck]

Here's how I set up my buffers. In my code I only use lfs_file_opencfg with &file_config as the last argument.

lfs_t lfs;
lfs_file_t lfs_file;
uint8_t read_buffer[WB_PAGE_SIZE];
uint8_t prog_buffer[WB_PAGE_SIZE];

// configuration of the filesystem is provided by this struct
const struct lfs_config lfs_cfg = {
    // block device operations
    .read = _read_littlefs_wrapper,
    .prog = _write_littlefs_wrapper,
    .erase = _erase_littlefs_wrapper,
    .sync = _sync_littlefs_wrapper,

    // block device configuration
    .read_size = WB_PAGE_SIZE,
    .prog_size = WB_PAGE_SIZE,
    .block_size = WB_BLOCK_SIZE * WB_PAGE_SIZE,
    .block_count = WB_BLOCK_COUNT,
    .cache_size = WB_PAGE_SIZE,     // storing reads in RAM
    .read_buffer = read_buffer,
    .prog_buffer = prog_buffer,
    .lookahead_size = WB_PAGE_SIZE, //
    .block_cycles = 100000,         // from winbond datasheet -> wear leveling might not be an issue here
    //TODO Can try reducing the metadata_max size to help with 
    //the speed of metadata compaction - hopefully speed up 
    //.metadata_max = WB_PAGE_SIZE    //make metadatamax 1 page rather than 1 whole block  
    //can also try reducing read_size
};

// must be equal to cache size in lfs config
uint8_t file_buffer[WB_PAGE_SIZE];
struct lfs_file_config file_config = {
    .buffer = file_buffer};

The error is being thrown during lfs_mount()

The code I'm calling is included below.

int8_t lfs_init() {
  //initialize spi for memory chip
  bool flash_init = wb_init();

  wb_spi_start();
  // note: LFS will be mounted once and remain mounted throughout program operation
  int m_err = lfs_mount(&lfs, &lfs_cfg);

  // reformat the memory if we can't mount the filesystem
  // this will happen for any new or erased chip
  if (m_err<LFS_ERR_OK) {
    lfs_format(&lfs, &lfs_cfg);
    m_err = lfs_mount(&lfs, &lfs_cfg);
    if(m_err != LFS_OK)
       return m_err;
  }
  #if (DBG_NERV_LITTLEFS)
  NRF_LOG_INFO("LFS Mounted Successfully");
  NRF_LOG_FLUSH();
  #endif
  return 1;
}

[geky]

Ah! I forgot about the lookahead buffer.

Thanks for the code, reproducing locally forced me to realize my mistake.

The lookahead buffer is an additional buffer used to keep track of free blocks between block allocation scans. It needs to be backed by memory. The exact size is somewhat arbitrary, larger=fewer allocation scans, smaller=less ram.

NOTE: The lookahead buffer needs to be 32-bit aligned and a multiple of 8.

// note the lookahead buffer needs to be 32-bit aligned and a multiple of 8
// note lookahead_size is in units of bytes, but our array needs to be uint32s for alignment
uint32_t lookahead_buffer[LOOKAHEAD_SIZE / 4];

// in struct lfs_cfg
.lookahead_buffer = lookahead_buffer,
.lookahead_size = LOOKAHEAD_SIZE,

My bad. I will update the my original comment to be correct.