Optimize Blobs and Files in Dexie Cloud

[dfahlander]

Storing images, movies, tunes, sounds and files in IndexedDB is best accomplished by storing the data in Blob properties.

When using Dexie Cloud, Blobs are currently supported but they are stored in the SQL database on the backend in Base64 format. There are a few downsides with this:

• Table rows on the server are not optimized for large strings
• It would consume more storage than needed as it is converted to base64
• It would consume more bandwidth than needed
• It would consume more CPU power than needed
• The sync flow could be slow and it could take too long time to download the blobs on sync

This feature request is to optimize Blobs and Files specifically and store them in a Blob storage cloud service rather than the SQL database on the server. Since Blob and File are immutable, we won't need any specific support for updating contents within a Blob. If a Blob based property is changed, this will always lead to a new Blob being created.

Updated sync flow:

• [ ] Client side sync (push): upload Blobs to a dedicated endpoint that returns the blob location. Just sync the blob location and not the content to cloud.
• [ ] Server sync endpoint (push): Just store the blob location in the SQL database.
• [ ] Server sync endpoint (pull): Generate signed URL for blobs
• [ ] Client side sync (pull): Construct blobs from the signed URLs and store them in Dexie.

Server side CRUD operations

• [ ] create: Just add the blob location
• [ ] read: Retrieve a signed URL for the blob location and return to the client
• [ ] delete: Delete the blob from the blob storage
• [ ] update: Delete the old blob from blob storage. Just add the new blob location

Client side CRUD operations

• not affected

[dusty-phillips]

Suggestion: hash blobs such that if multiple copies of the same image are in a user’s db (maybe in different realms), you only have to store and sync it to the device once.

[dfahlander]

Suggestion: hash blobs such that if multiple copies of the same image are in a user’s db (maybe in different realms), you only have to store and sync it to the device once.

Thanks, it's probably a good idea but it puts some complexity in: If someone deletes a blob or even a whole DB that has references to it, we'd need to keep a ref-count to know when it would be safe to delete it. So I probably start out without this and add it later... Thanks for the suggestion!