search

littlefs-project / littlefs

A little fail-safe filesystem designed for microcontrollers
BSD 3-Clause "New" or "Revised" License
5.17k stars 794 forks source link

[documentation request] limits and constrains #738

Open atesin opened 2 years ago

atesin commented 2 years ago

there is no clear, centralized and official info about LittleFS limits... for example:

geky commented 2 years ago

Hi @atesin, thanks for creating an issue. I can answer most of these though point taken that more documentation is needed. Setting up and filling out a wiki is on my TODO list.

  • accepted chars for filenames/dirnames, CHARACTER SET and constrains
  • valid file/dir names format, reserved file/dir names

I just left a comment here that may be useful: https://github.com/littlefs-project/littlefs/issues/729. LittleFS follow Linux/Unix, so all characters except '/' and NULL are allowed, and the only reserved filenames are '.' and '..'.

  • max filename/dirname lenght

This is configurable at format time: LFS_NAME_MAX. By default it's 255 bytes, and can be extended up to 1022 bytes. Though it's just as common to decrease it as this changes the size of the lfs_info struct.

  • max path lenght

This is unbounded except for I guess your device's memory. LittleFS never stores the full path.

  • max subdirectory depth

This is also unbounded for the same reason.

  • max number of files per filesystem/dir/rootdir/etc

All of these are unbounded except for how much you can store on your disk. The directories grow as needed.

  • max file size (related to blocksize?)

This is also configurable at format time: LFS_FILE_MAX, though I don't know of a reason to change it. It's limited to 32-bits on disk, and 31-bits in the current driver, so ~2GiB.

  • min/max filesystem size (related to blocksize?)

LittleFS uses 32-bit block addresses, so ~2Giblocks. The actual limit in bytes depends on your block size, which can be a multiple of the physical block size similar to FAT's "cluster size". So if you used 512 byte blocks the limit would be ~1TiB, if you upped this to 4KiB blocks the limit would be ~8KiB, etc. I suppose at a max of ~2GiB block size you could reach a limit of 4 EiBs but that would be a bit silly.

It's worth noting LittleFS currently has some performance issues that cause real problems for larger filesystems. This is something I'm working on, but the state of scaling performance is not that great.

  • min/max block size, increments, etc

Block size is also 32-bits, so the theoretical maximum is ~2GiB. The theoretical minimum is ~128 bytes, due to the upper limit of prefixed metadata needed by the CTZ skip-list [explanation]. But a more practical minimum is 512 bytes. This is because attributes and file names are limited by block size, trying to store a 256 byte file name on a 128 block-size filesystem will lead to unintentionally confusing errors. ~In the future if there is a major revision with breaking changes 512 bytes will become a hard minimum. I haven't seen much motivation for smaller block sizes.~

I'm not sure what "increments" means here but if elaborate I can probably answer.

e107steved commented 2 years ago

"I haven't seen much motivation for smaller block sizes." Not sure whether you're considering block sizes smaller than 128 bytes, or smaller than 512 bytes, in this comment. One of the reasons I currently use littleFS (apart from its robustness) is that it has a very small fixed overhead of "disc" space for root directory and the like - unlike FAT which needs, IIRC, tens, and maybe hundreds, of kb. I'm using it with relatively small FRAM devices to store configuration files and the like - currently 128kb, although littleFS is very viable with as little as 32K of NVM. In this situation 128 byte block size works well (and probably no point going smaller); 512 byte blocks could maybe cause some problems (mitigated to some extent by inline file storage of small files).

geky commented 2 years ago

Oh, thanks for commenting. FRAM is an interesting case where 128-bytes would make sense. Have you seen any metadata-related issues from this block-size?

128-byte blocks would also help reduce the overhead of smallish files that aren't inline-able. This probably deserves more investigation and better testing/conditions around the name/attrs-bigger-than-block-size case. It would be interesting to look at the total storage utilization wrt to small block sizes.

atesin commented 2 years ago

@geky : thanks for your interest

in respect to "I haven't seen much motivation for smaller block sizes."... i am trying (learning) to use LittleFS in arduino (esp8266 actually) with a little IoT application.... my app will store files with about 20 or 30 bytes (just a couple short arrays and single word strings), in a tiny eeprom filesystem with 1-2mb maybe? (even smaller in other platforms, like arduino uno with 1kb!) .... so i think storing them in a 512 bytes block size will be a huuuge waste of space

in respect what i mean with sizes and "increments" .... i expected a response like "minimum 64 bytes, maximum 1024 bytes, with increments of 32 bytes" (i mean 64, 96, 128, 160, 192... 992, 1024)

would be great if in the future you could synthesize this info and include it in documentation, or in a wiki page for example, when you can... so more people could consult this info, is important when designing apps..... for example i had huge doubts about files count, sizes and names, and after reading a lot and thinking i finally take the risk and used 128 chars path length and this pattern for file names validation, i hope won't conflict:

e107steved commented 2 years ago

Think I started by trying 64-byte blocks, but didn't work, so moved to 128-byte blocks. I'm sure there's a thread here somewhere where you confirmed that 128-byte blocks were the minimum, due to metadata. Certainly not seen any problems; its just worked. These are the settings I've used:

define LFS_I2C_READ_BLOCK_SIZE 1

define LFS_I2C_WRITE_BLOCK_SIZE 128

define LFS_I2C_ERASE_BLOCK_SIZE 128

define LFS_I2C_LOOKAHEAD_BLOCK_COUNT 128

define LFS_I2C_CACHE_SIZE 128

IIRC the minimum read for most littleFS operations is 16 bytes, but don't think it made any difference increasing LFS_I2C_READ_BLOCK_SIZE