The specification should say how large an Earthstar document is allowed to be. It's not specified right now!
More precisely, how long is the content field allowed to be.
Scuttlebutt's message limit is something like 8 kb. I think we could have something much larger, maybe 3 to 5 mb, because:
we don't have a separate blob protocol
we can delete messages so we're less concerned about storage space
we will eventually have selective sync using sync queries #6 which will let you only ask for documents you care about
we will eventually have sparse mode #33 for handling document metadata without also downloading the data itself up front
Let's choose a size that's useful for most individual files in a web context (javascript, images, etc) so people don't have to mess around with breaking files into chunks.
[x] Choose a size
[x] Write it in the spec, here. Make sure we specify length in bytes and not the number of unicode code points
[ ] Consider bumping the format from es.4 to es.5, unless this is considered a bugfix, which it might be
[x] Implement a check for content length in ValidatorES4
[ ] Syncing code can also use this to avoid ever trying to JSON.parse an enormous string, but it will need to add some safety margin to account for the other parts of a Document besides the content.
The specification should say how large an Earthstar document is allowed to be. It's not specified right now!
More precisely, how long is the
content
field allowed to be.Scuttlebutt's message limit is something like 8 kb. I think we could have something much larger, maybe 3 to 5 mb, because:
Let's choose a size that's useful for most individual files in a web context (javascript, images, etc) so people don't have to mess around with breaking files into chunks.
format
fromes.4
toes.5
, unless this is considered a bugfix, which it might becontent
.We already have a function
stringLengthInBytes
which counts the number of bytes needed to store a string after encoding it to UTF-8: https://github.com/earthstar-project/earthstar/blob/a19ff805b2739b423fc75edc178eb2dec58ff488/src/storage/query.ts#L186-L187