A potential user on Discord was asking about general purpose file storage within BonsaiDb. BonsaiDb has a 4GB document size limit, which comes from Nebari's "chunk size" limit. More importantly, however, Nebari treats chunks as whole pieces of information that must be loaded completely in memory to be returned. The side effect is that while BonsaiDb has a 4GB document size limit, the practical limit is the available RAM.
Even if Nebari's limit were increased, it would require so much RAM, it wouldn't be practical.
MongoDB offers an easy abstraction that allows storing larger files in segments. CouchDB supports an attachments API which allows associating one or more blobs of data with a document. Both of these ideas seem like good starting points for considering how to enable handling of large blobs, including the ability to stream the blobs without loading the entire blob in to memory.
A potential user on Discord was asking about general purpose file storage within BonsaiDb. BonsaiDb has a 4GB document size limit, which comes from Nebari's "chunk size" limit. More importantly, however, Nebari treats chunks as whole pieces of information that must be loaded completely in memory to be returned. The side effect is that while BonsaiDb has a 4GB document size limit, the practical limit is the available RAM.
Even if Nebari's limit were increased, it would require so much RAM, it wouldn't be practical.
MongoDB offers an easy abstraction that allows storing larger files in segments. CouchDB supports an attachments API which allows associating one or more blobs of data with a document. Both of these ideas seem like good starting points for considering how to enable handling of large blobs, including the ability to stream the blobs without loading the entire blob in to memory.