nostr-protocol / nips

Nostr Implementation Possibilities
2.19k stars 519 forks source link

Event Compression Feasibility #265

Open alltheseas opened 1 year ago

alltheseas commented 1 year ago

Please bear with my non-technical nature & understanding of nostr. In a former professional life the ability to compress certain data types made a lasting impression on me.

Per my limited understanding, it seems NIP-01 describes JSON.

NIP-65 talks about "compressed form", and I understand this to be a different type of compression (i.e. relaying less info/in more compact form).

Doing a quick search, JSON size appears to be compressible by ~75% per https://tukaani.org/lzma/benchmarks.html.

Is it feasible technically to have relay + client compress, and uncompress events, thereby saving bandwidth? For small event size, might this make sense and benefit client + relay?

If yes, what might the tradeoffs be (for instance, increased compute, and therefore battery drain)?

barkyq commented 1 year ago

There is an accepted compression extension called permessage-deflate for transmitting data through websockets, see: https://www.rfc-editor.org/rfc/rfc7692#section-7

Both the client and relay need to implement permessage-deflate in order for the compression to be used. Some relays do not implement it (yet), but I know for a fact that knostr and strfry relay do implement it.

It allows pretty good compression for nostr messages, sometimes hitting around your desired 75%. The compression is best when there is a lot of repetition between returned messages (often there is repetition, as subscription ids, pubkeys, etc are repeated across many messages in a row).

EDIT: shameless plug. finally my relay implementation gnost-relay also has it.

alltheseas commented 1 year ago

Orange crush mentions that strfry compresses relay-to-relay communication, however strfry does not compress relay-client data.

https://damus.io/note1e5xxzgt0wr7ylnfnv99pd8zu29ya8lrj66vknuwdklu0p59f7plqqg9vmn

OC suggests yesstr for client-relay comms.

hoytech commented 1 year ago

No, strfry does compress relay->client and client->relay communications by default (over both regular nostr and yesstr). It uses per-message sliding-window compression, if client supports it. I wrote a little bit more about this here: https://github.com/hoytech/strfry/issues/22#issuecomment-1429760385

barkyq commented 1 year ago

little CLI go program for testing compression: https://github.com/barkyq/gnost-deflate-client