Bad behavior with append-only files (logs)

[rojer]

I want to keep a log file on littlefs. The file is opened once at boot and then appended to, then closed and a new one is opened. I'm running into multiple issues.

• While the file is open (read: all the time), size is reported as 0 by stat.
• If device is powered off while file is opened (read: all the time), LFS is unable to recover and the file becomes unwritable. I don't have time to produce detailed diagnostics, just throwing it out there and I'll have to abandon LFS.

Point is, this case needs more attention and testing.

[geky]

It sounds like you need to add calls to lfs_file_sync. littlefs won't commit data to disk unless explicitly asked to (lfs_file_sync or lfs_file_close). Writing to a file stages data but doesn't commit it to disk. This is also the reason lfs_stat is still reporting 0.

The goal is to help users by preventing half-written files from showing up after a power-loss.

I realize this is not documented as well as it could be.

---

The 0-length file is the default state for new files. It's needed because we need to create a directory entry to store the filename so we don't have to keep it in RAM. I've considered removing this by adding some sort of "not written" flag, but haven't been able to implement it yet.

[seregega]

Confirmed. Sync file after write can help to save the data in this case.

[M-Bab]

During extensive testing I recognized sometimes an fsync is not enough to have all data actually written. Because we are working with small blocks of data (64 bytes), I had to artificially flood the buffer when I really want to have all data written:

void LOG_QuickSync(bool bFloodBuffer)
{
  if (logFileWriteHandle != NULL)
  {
    if (bFloodBuffer)
    {
      /* Fill Cache with enough data that definitely all events are written. */
      static const uint8_t DataBufferFiller[CONFIG_LITTLEFS_WRITE_SIZE] = {0};
      fwrite((void *) DataBufferFiller, sizeof(uint8_t), sizeof(DataBufferFiller), logFileWriteHandle);
      ESP_LOGI(TAG, "Forced quick sync for up-to-date log done!");
    }
    fsync(fileno(logFileWriteHandle));
  }
}

All tests with different littlefs configurations (changed PAGE, READ, WRITE, LOOKAHEAD and CACHE size) did not help. So I am posting this here that it might resolve the problem for others - or maybe someone comes up with a solution that the workaround is not needed.