threefoldtech / rfs

A fuse filesystem in rust
Apache License 2.0
1 stars 1 forks source link

improve `dir` store for performance #52

Open muhamadazmy opened 3 months ago

muhamadazmy commented 3 months ago

Right now dir store stores all the blobs flat under the store dir path. (all blobs in one directory) which makes the filesystem slow (too much files)

A better layout is to create 2 levels as follows

/ 
  - [XX] / XX...
  -           XX...

Simply use the fist byte of the hex as a directory name, so we have subdirs with 00, 01, 02 ... FF as a sub directory then each dir of these will then has only the blobs that are prefixed with that dir name

IMPORTANT: make sure that on get the store can still use this format and the old format for compatibility