MarFS provides a scalable near-POSIX file system by using one or more POSIX file systems as a scalable metadata component and one or more data stores (object, file, etc) as a scalable data component.
(as opposed to MDAL.) This will be used to support MC Striped Repos.
Our data-storage is (and is expected to continue to be) done through RESTful semantics: GET/PUT/DELETE. Our object-streams provide operations like stream_open(), stream_write(), stream_close(). These are sequential operations to a stream, with a curl PUT underneath. (And there's similar for GET, DELETE, etc). The DAL doesn't require anything beyond that, though it will be writing to NFS-mounted remote file-systems underneath.
Where the MDAL looks POSIXy, the DAL should just look like our object-streams support. The existing object-streams functions would just plug right into the DAL function-ptrs. Then we'll write new alternative implementations to support the Multi-Component Striped Repos stuff we've been talking about recently.
Some minor changes to object-IDs are needed so the slashes don't imply a directory hierarchy. Simplest start might be to just change '/' to something else. Then object-IDs get hashed to select the "server" mount-point, and hashed again to select the leaf in the scatter-tree of subdirs under that.
(as opposed to MDAL.) This will be used to support MC Striped Repos.
Our data-storage is (and is expected to continue to be) done through RESTful semantics: GET/PUT/DELETE. Our object-streams provide operations like stream_open(), stream_write(), stream_close(). These are sequential operations to a stream, with a curl PUT underneath. (And there's similar for GET, DELETE, etc). The DAL doesn't require anything beyond that, though it will be writing to NFS-mounted remote file-systems underneath.
Where the MDAL looks POSIXy, the DAL should just look like our object-streams support. The existing object-streams functions would just plug right into the DAL function-ptrs. Then we'll write new alternative implementations to support the Multi-Component Striped Repos stuff we've been talking about recently.
Some minor changes to object-IDs are needed so the slashes don't imply a directory hierarchy. Simplest start might be to just change '/' to something else. Then object-IDs get hashed to select the "server" mount-point, and hashed again to select the leaf in the scatter-tree of subdirs under that.