Verify file read is not corrupted

[KRekha3]

i want to read file crc to verify whether file corrupted or not for that i need to check all the CRC values written in each blocks. is this has been taken care in current lfs.c code ? does lfs_file_open() checks for all the CRC values before opening file ? if not what is the best way to verify file corruption?

[geky]

Hi @KRekha3, thanks for creating an issue.

Currently littlefs does not CRC file data, only the file's metadata (size, name, any attributes, etc).

I've gone back and forth a bunch on whether or not littlefs should CRC the actual file data. It's not strictly necessary for power-resilience and comes with a performance cost. It raises the question do we check the CRC on every read or just once in open? On top of that most devices that risk data corruption will have ECC at the block device level.

But it shouldn't be too hard to provide your own CRC. There's two option, either provide a CRC at the block device level or at the file level.

Feel free to call lfs_crc directly, it's a part of the public API. Note that 32-bit CRCs are considered only good for up to 512 MiB.

1. CRC on block device

   littlefs does not require exact power-of-2 block sizes. This means if you are clever with your block device implementation you can sneak a CRC into every block.

   littlefs will respect LFS_ERR_CORRUPT errors on block device reads, however these are just passed directly to the user as littlefs does not have a way to recover if the file data is corrupted (this would require duplicating file data).

2. CRC on file

   A different option, which may or may not be easier depending on what file operations you need, is to apply a CRC on top of the file. This is easiest if you only ever write the file all at once or as a stream.

   littlefs has what it calls "custom attributes", which are small <~1KiB attributes associated with each file. This would be the perfect place to store a file's CRC. littlefs also supports writing out these file attributes atomically when you update the file.

   #define FILE_CRC_ATTR 0x11 // arbitrary
   uint32_t file_crc;
   struct lfs_attr file_attrs[1] = {{
      .type = FILE_CRC_ATTR,
      .buffer = &file_crc,
      .size = sizeof(file_crc),
   }};
   struct lfs_file_config file_config = {
      .attrs = file_attrs,
      .attr_count = 1,
   };
   lfs_file_t file;
   
   // open file, this will also load the file's attributes
   lfs_file_opencfg(&lfs, &file, "filepath", LFS_O_RDWR, &file_config);
   
   // read and check crc
   uint8_t buffer[256];
   lfs_file_read(&lfs, &file, buffer, 256);
   uint32_t temp_crc = lfs_crc(0xffffffff, buffer, 256);
   LFS_ASSERT(temp_crc == file_crc);
   
   // write and update crc
   file_crc = lfs_crc(0xffffffff, buffer, 256);
   lfs_file_write(&lfs, &file, buffer, 256);
   
   // attributes and data are not written to disk until file close or sync
   lfs_file_close(&lfs, &file);

   You can also use lfs_getattr, lfs_setattr, and lfs_removeattr to access attributes directly, but these are not atomic.

File-level CRCs is probably the easiest solution, but it depends on what you are doing to the files. If you are doing a lot of random reads/writes managing the file's CRC may get complicated.

Hopefully that helps, but let us know what other questions you run into.

[KRekha3]

Thank you @geky . Your detailed explanation helped me in finding better solution. I will try to implement with File CRC.