search

mozilla-services / pushbox

Durable Storage for Push
Mozilla Public License 2.0
6 stars 9 forks source link

Actually "large" payloads: data only API? #69

Closed ancoron closed 4 years ago

ancoron commented 4 years ago

The main goal states the following:

[...] mozilla needs the ability to store large data chunks that may not fit perfectly within a standard WebPush message [..]

So, as I interpret "large" might go into the MiB's to GiB's size region, for which it would be unbearable for any reasonably performing client to require wrapping of the actual data content into a JSON, as the current API requires:

{"ttl": 3600, /* Time for the data to live in seconds.*/
 "data": "aBc1..." /* Encrypted data block to store */
}

The same issue exists for downloading the data, even more so as it represents a list of uploaded entities.

Is there any plan or brainstorming for an alternative that allows to use headers to provide the metadata (e.g. to move the TTL into a HTTP header and just have the binary encrypted content as payload)?

That would also allow to "stream" up- and downloads, further reducing client-side resources and improving server-side scalability.

jrconlin commented 4 years ago

1) Push currently limits to around 4KB. (around 3KB for mobile because of transcoding issues when using APNs, FCM, etc.)

2) Pushbox is kind of a hack around that limit so we could send data that can be slightly more than 4KB. (Think "Send Tab to Device" which uses sync encrypted data blobs and may carry some additional metadata that may well exceed the 3KB limit. Some of those URLs are really long.)

3) This is an internally used service, which means there are actual fiscal costs tied to it that absolutely limit what we can do. Seriously. As in "why are there so many zeroes?" kind of bills.

Honestly? If we wanted to be super-efficient here, we'd make a customized packet type that delivered the data over the existing WebPush websocket connection directly to the sync service using a set of customized, proprietary APIs and drop this repo entirely. That approach would really complicate an already fairly complicated set of systems and give our SecOps folk nightmares, so we will probably not do that.

Granted, if your "Spiffy-Cool v.2.0" service could take advantage of whatever it is we're doing here, go for it! Fork, modify, and optimize like the mad wizards you are.

So, to your questions: No, we've not really thought about that since we built this hack, and no, we'll probably not do that since we're not going to be sending massive payloads over this service.