The S3 storage system relies on the local relational database as a single point of failure for representing the file tree, including file names and their mappings to payload objects in the S3 bucket.
Thus, the resilience of S3 storage for file payloads is offset by any vulnerability in the application and especially database deployment. Database backup is currently essential for disaster recovery, but still extremely clumsy as a solution for extracting file trees or file contents directly from S3 storage.
Requested is periodic updates to S3 storage of a representation of file tree data, sufficient to permit reconstruction of each user's file tree from only the object storage in the bucket.
If the object data would be structured transparently, then recovery by standalone utilities may be feasible, in case administrators need to extract file contents without reconstituting a full deployment of the application. In principle, an object referenced by a single prescribed key may serve as a manifest for the contents of the entire bucket.
The concept opens opportunities for further enhancements, such as complete replication of all site data into an S3 store for purposes of disaster recovery, as well as seamless site migration.
joshtrichards
commented
1 year ago
The S3 storage system relies on the local relational database as a single point of failure for representing the file tree, including file names and their mappings to payload objects in the S3 bucket.
Thus, the resilience of S3 storage for file payloads is offset by any vulnerability in the application and especially database deployment. Database backup is currently essential for disaster recovery, but still extremely clumsy as a solution for extracting file trees or file contents directly from S3 storage.
Requested is periodic updates to S3 storage of a representation of file tree data, sufficient to permit reconstruction of each user's file tree from only the object storage in the bucket.
If the object data would be structured transparently, then recovery by standalone utilities may be feasible, in case administrators need to extract file contents without reconstituting a full deployment of the application. In principle, an object referenced by a single prescribed key may serve as a manifest for the contents of the entire bucket.
The concept opens opportunities for further enhancements, such as complete replication of all site data into an S3 store for purposes of disaster recovery, as well as seamless site migration.