search

remotehack / remotehack.github.io

https://remotehack.space
MIT License
7 stars 3 forks source link

compress API data #60

Open javorszky opened 4 years ago

javorszky commented 4 years ago

You know how most good APIs communicate by sending and receiving json?

json is kinda verbose and super compressable, so what if:

we compress the data first!

sender -> json -> compression middleware -> (internet) compressed binary -> decompression middleware -> json -> receiver

or, better yet

yeet the json data into a jpeg image

because nothing says "beautiful data" like an image <3

or, as a third experiment

generate some soundform

throwback to the modem screams of the 90s!

javorszky commented 4 years ago

I have been told this exists: https://msgpack.org/index.html

nevertheless the jpeg / mp3 screaming shall commence

benfoxall commented 4 years ago

Ha!! Audio endpoints sound awesome! I've done a bit with the web audio api if that's helpful at all 🎵

spikeheap commented 4 years ago

I wonder if we could interleave the data into an existing soundform (like the podcast episodes we'll be recording...). Gotta love a bit of steganography. That goes a little bit against the optimisation theme, but 🤷‍♂️.

The less fun version of JSON compression is built into browsers by us using gzip/brotli/etc. using Accept-Encoding headers (though you still end up needing middleware server-side to compress outgoing data). Modern compression algorithms are great with repetition too, so we can worry less about the verbosity of the JSON unless we're in real-time/super-high-throughput territory. That said... it doesn't look like compression is applied as often as I initially believed. It could be interesting to hack a comparison of optimised format like msgpack with basic compression too.

It's too early for me to run Lighthouse over RemoteHack 😉, but I like that they call out uncompressed resources too.

Edit: something like this might be fun to try.

peterjwest commented 4 years ago

I love the idea of lossy compression of JSON 😂. At the other end you could write a fixerer that tries to make it work again.

benfoxall commented 4 years ago

If you had a single endpoint and use Accept header to call different transformers, it could be a really fun project for people to implement their own schemes:

GET /content
Accept-Encoding: gzip, deflate

…
Accept-Encoding: jpg

…
Accept-Encoding: mp3

…
Accept-Encoding: shakespearean_sonnets