This PR adds support for replicas to hold documents of many formats; a new document format es.5 which can have binary data of any size associated with it; and support for setting, retrieving and syncing this associated data, hereafter referred to as attachments.
Whereas es.4 documents could hold a maximum of 4mb, there is no limit for attachment sizes.
FormatEs5
This is a successor to es.4 with a few key differences:
content has been replaced with text
The contents of the text field can be a maximum size of 8kb (not 4mb as with es.4)
workspace has been renamed to share
es.5 documents can have attachments, possibly large amounts of associated binary data. When a es.5 document has an attachment, it has attachmentHash and attachmentSize fields, and the path must have a file extension, e.g. /music/song.mp3. Only documents with attachments can have a file extension.
These documents can be synced sparsely, that is they can be shared around with or without their attachments. It's up to peers to decide if they want a docs attachments or not.
Additionally, IFormatValidator is now IFormat, and has taken on a few new responsibilities: it has methods for generating new docs in addition to simply validating them, and new methods for getting information about a document's attachments.
Replica
.set now defaults to setting es.5 documents, which can take attachments in the form of UInt8Array or ReadableStream<UInt8Array>. Providing an optional format argument lets you set docs of other formats.
Added .getAttachment, which takes a doc and returns a DocAttachment for it when possible.
Added addAttachments which takes an array of docs and adds an attachment field to each of them with a DocAttachment, undefined (there's an attachment but it's not locally available), or ValidationError (doc can't have an attachment).
Added ingestAttachment, used when setting and syncing document attachments.
Added wipeDocAtPath, which wipes a documents contents and deletes any associated attachments.
Now deletes all expired docs and dangling attachments upon the replica closing, as well as every hour.
getAllDocs, getLatestDocs, getAllDocsAtPath, getLatestDocAtPath, queryDocs, queryPaths, queryAuthors, overwriteAllDocsByAuthor, getQueryStream all take an optional argument for the desired document format. Defaults to FormatEs5.
getMaxLocalIndex is now async
Replica drivers
Replica drivers are now composed of two parts: a doc driver and an attachment driver. These have their own interfaces, IReplicaDocDriver and IReplicaAttachmentDriver.
Updated the following drivers to support docs of many formats: DocDriverSqlite (Deno + Node), DocDriverSqliteFFI, DocDriverIndexedDB, DocDriverMemory, and DocDriverLocalStorage.
DocDriverIndexedDB no longer dumps all of a shares documents into memory, but uses IndexedDB like a real database.
Added AttachmentDriverFilesystem (Deno, and a Node version contributed by @achou11!)
Added AttachmentDriverIndexedDB
Added AttachmentDriverMemory
Syncing
Syncing now transfers attachments between peers as well as docs.
Added attachment transfer statuses to getStatus and the status from onProgress callback.
ISyncPartner interface has been updated with new methods for syncing attachments.
Other
Crypto: Added updatableSha256 method.
Removed filter.contentLengthGt from Query.
Added AttachmentStreamInfo, which implements TransformStream for ReadableStream<Uint8Array> and is able to calculate the hash and given size of the stream's contents.
This PR adds support for replicas to hold documents of many formats; a new document format
es.5which can have binary data of any size associated with it; and support for setting, retrieving and syncing this associated data, hereafter referred to as attachments.Whereas
es.4documents could hold a maximum of 4mb, there is no limit for attachment sizes.FormatEs5
This is a successor to
es.4with a few key differences:contenthas been replaced withtexttextfield can be a maximum size of 8kb (not 4mb as withes.4)workspacehas been renamed tosharees.5documents can have attachments, possibly large amounts of associated binary data. When aes.5document has an attachment, it hasattachmentHashandattachmentSizefields, and the path must have a file extension, e.g./music/song.mp3. Only documents with attachments can have a file extension.These documents can be synced sparsely, that is they can be shared around with or without their attachments. It's up to peers to decide if they want a docs attachments or not.
Additionally,
IFormatValidatoris nowIFormat, and has taken on a few new responsibilities: it has methods for generating new docs in addition to simply validating them, and new methods for getting information about a document's attachments.Replica
.setnow defaults to settinges.5documents, which can take attachments in the form ofUInt8ArrayorReadableStream<UInt8Array>. Providing an optional format argument lets you set docs of other formats..getAttachment, which takes a doc and returns aDocAttachmentfor it when possible.addAttachmentswhich takes an array of docs and adds anattachmentfield to each of them with aDocAttachment,undefined(there's an attachment but it's not locally available), orValidationError(doc can't have an attachment).ingestAttachment, used when setting and syncing document attachments.wipeDocAtPath, which wipes a documents contents and deletes any associated attachments.getAllDocs,getLatestDocs,getAllDocsAtPath,getLatestDocAtPath,queryDocs,queryPaths,queryAuthors,overwriteAllDocsByAuthor,getQueryStreamall take an optional argument for the desired document format. Defaults to FormatEs5.getMaxLocalIndexis now asyncReplica drivers
Replica drivers are now composed of two parts: a doc driver and an attachment driver. These have their own interfaces,
IReplicaDocDriverandIReplicaAttachmentDriver.Updated the following drivers to support docs of many formats:
DocDriverSqlite(Deno + Node),DocDriverSqliteFFI,DocDriverIndexedDB,DocDriverMemory, andDocDriverLocalStorage.DocDriverIndexedDBno longer dumps all of a shares documents into memory, but uses IndexedDB like a real database.Added
AttachmentDriverFilesystem(Deno, and a Node version contributed by @achou11!)Added
AttachmentDriverIndexedDBAdded
AttachmentDriverMemorySyncing
Syncing now transfers attachments between peers as well as docs.
getStatusand the status fromonProgresscallback.ISyncPartnerinterface has been updated with new methods for syncing attachments.Other
updatableSha256method.filter.contentLengthGtfromQuery.TransformStreamforReadableStream<Uint8Array>and is able to calculate the hash and given size of the stream's contents.