The goal of this pull request is to offer a general purpose approach to storing files within BonsaiDb. I also see this as a simpler playground for me to experiment with how to handle API permissions than the jobs pull request.
After pondering how to make this as useful as possible while keeping it fairly simple, here's the approach I've decided to take:
This is a new crate, bonsaidb-files. Internally it defines two collections: File and Block. The File collection contains all the information to build a virtual filesystem: Name, Path, Metadata.
The public facing API exposes a File type that accepts a generic parameter: a FileConfig implementation. This trait allows customizing the collection names and the BLOCK_SIZE.
When appending data to a file, the data is broken into chunks with a maximum size of BLOCK_SIZE. The current implementation uses 64kb as the block size.
The file contents can be queried, at which point a Contents type is returned. Internally, this type has a list of all of the block IDs and their lengths.
The Contents type will implement:
[x] std::io::Read
[x] std::io::Seek
[x] A way to append to the file, buffered using std::io::Write
[x] tokio::io::AsyncRead
[x] A way to append to the file, buffered using tokio::io::Write
Behind the scenes, the Contents type supports requesting blocks in chunks from the database. This defaults to 10, but can be customized.
Operations planned to be supported:
[x] Create Directory
[x] List directory contents
[x] Walk directory
[x] Truncate file (both from start and from end).
[x] Delete
[x] Move file
[x] Async versions of all operations
Other Todos:
[x] Validate file names
Ideas maybe worth exploring:
[x] Instead of "batching_by_blocks", maybe it should just be a "memory_limit" parameter which can be divided by the block size. This also would help with anyone who is appending small chunks, as those will be their own blocks.
Closes #222
The goal of this pull request is to offer a general purpose approach to storing files within BonsaiDb. I also see this as a simpler playground for me to experiment with how to handle API permissions than the jobs pull request.
After pondering how to make this as useful as possible while keeping it fairly simple, here's the approach I've decided to take:
This is a new crate,
bonsaidb-files. Internally it defines two collections:FileandBlock. TheFilecollection contains all the information to build a virtual filesystem: Name, Path, Metadata.The public facing API exposes a
Filetype that accepts a generic parameter: aFileConfigimplementation. This trait allows customizing the collection names and theBLOCK_SIZE.When appending data to a file, the data is broken into chunks with a maximum size of
BLOCK_SIZE. The current implementation uses 64kb as the block size.The file contents can be queried, at which point a
Contentstype is returned. Internally, this type has a list of all of the block IDs and their lengths.The
Contentstype will implement:std::io::Readstd::io::Seekstd::io::Writetokio::io::AsyncReadtokio::io::WriteBehind the scenes, the
Contentstype supports requesting blocks in chunks from the database. This defaults to 10, but can be customized.Operations planned to be supported:
Other Todos:
Ideas maybe worth exploring: