search

jondot / sneakers

A fast background processing framework for Ruby and RabbitMQ
https://github.com/jondot/sneakers
MIT License
2.24k stars 333 forks source link

Support content-encoding in worker/publisher #449

Closed ansoncat closed 2 years ago

ansoncat commented 3 years ago

Similar to the content-encoding header in HTTP, ContentEncoding implemented here provides a way to encode/decode (compress/decompress) messages on the fly. Like original ContentType, ContentEncoding can be configured like:

Sneakers::ContentEncoding.register(
  content_encoding: 'gzip',
  decoder: ->(payload) { ActiveSupport::Gzip.decompress(payload) },
  encoder: ->(payload) { ActiveSupport::Gzip.compress(payload) },
)

When messages arrive with the given content type and content encoding set the payload will be decoded and then deserialized. Before publishing, message will be serialized and then encoded with given content_type and content_encoding. Most of code are borrowed from ContentType (#152), thus behavior of configurations and options is pretty much the same.

With combination of ContentType and ContentEncoding, we can easily deal with something like 'text/plain'+'gzip', 'application/json'+'gzip' or 'application/json'+'zstd' without any duplicated code.

michaelklishin commented 2 years ago

Thank you. This is a non-trivial contribution I generally would like to see more feedback on. However, this fits the kind of features Sneakers strives to provide, and complements content type support nicely. So I think this should not receive a lot of pushback :)

michaelklishin commented 2 years ago

One thing this could do on the publisher side but does not is set the message property for content encoding, which AMQP 0-9-1 has. It also has one for content type and that isn't set either, which may be why this PR does nothing about it. The field is purely informative for applications and is not used by RabbitMQ itself in any way.

ansoncat commented 2 years ago

One thing this could do on the publisher side but does not is set the message property for content encoding, which AMQP 0-9-1 has. It also has one for content type and that isn't set either, which may be why this PR does nothing about it. The field is purely informative for applications and is not used by RabbitMQ itself in any way.

Thanks for merging code. On the publisher side, properties for content encoding and content type will be set automatically by bunny because we pass them as options to Bunny::Exchange::publish. Those options will be encoded as properties in here and then here.

michaelklishin commented 2 years ago

@ansoncat then they are not "set automatically by Bunny" but rather by your code ;) which is fine, of course.