MarFS provides a scalable near-POSIX file system by using one or more POSIX file systems as a scalable metadata component and one or more data stores (object, file, etc) as a scalable data component.
We need to be able to add more storage by adding a set of capacity units while still maintaining readability of the old data. We have discussed several schemes for this including distributing the objects uniformly at random on write, then stating all the capacity units in parallel to find the object for reading. This has the drawback that we can't easily re-balance objects across the available capacity.
I have discussed an alternative scheme based on consistent-hashing with Jeff that will allow for efficient background re-balancing (if wanted) and eliminates the need to stat every capacity unit. This scheme also facilitates the idea of "capacity targets" where an admin can decide we want to place 50% of the data on cap0 and 25% on cap1 and cap2 respectively.
We need to be able to add more storage by adding a set of capacity units while still maintaining readability of the old data. We have discussed several schemes for this including distributing the objects uniformly at random on write, then stating all the capacity units in parallel to find the object for reading. This has the drawback that we can't easily re-balance objects across the available capacity.
I have discussed an alternative scheme based on consistent-hashing with Jeff that will allow for efficient background re-balancing (if wanted) and eliminates the need to stat every capacity unit. This scheme also facilitates the idea of "capacity targets" where an admin can decide we want to place 50% of the data on
cap0and 25% oncap1andcap2respectively.