search

phoenixframework / phoenix_pubsub_redis

The Redis PubSub adapter for the Phoenix framework
175 stars 66 forks source link

Ability to set compression level for serialized payload term #51

Closed zoldar closed 4 years ago

zoldar commented 4 years ago

We are facing an issue with pretty big payloads coming from published mutations through Absinthe (essentially nested Ecto schema structs from database query results pushed through pubsub). Recently, the issue forced us to increase default redis output buffer limits (32 MB per client/node) to avoid waves of dropped connections due to rapid spike of larger payloads filling them faster than the clients are able to empty them.

Although reducing the size of the query results is definitely something to address eventually, one thing that would be really helpful to us is being able to compress the payloads before submission. :erlang.term_to_binary/2 provides compressed option, which seems like a good fit for the purpose (per initial tests, level 1 compression allows for 10x payload size reduction with relatively low cost in CPU time).

Would it be deemed useful to add such option to this adapter? Here's a proposal of the change that could allow setting this (defaults to no compression):

https://github.com/phoenixframework/phoenix_pubsub_redis/compare/master...sketch-hq:sketch-hq/add-compression

josevalim commented 4 years ago

An option is definitely welcome.

zoldar commented 4 years ago

@josevalim Great! Just for the record, there's https://github.com/phoenixframework/phoenix_pubsub_redis/pull/52 already that introduces this option.