TheClimateCorporation / squeedo

clojure core.async based amazon SQS message processing
Apache License 2.0
62 stars 28 forks source link

Add support for middleware when consuming messages #19

Open ghost opened 8 years ago

ghost commented 8 years ago

For each message I receive, I want to perform the following

Clojure Ring middleware provides great hooks for reducing the boilerplate required in order to follow this kind of workflow. It would be nice to be able to provide similar middleware for SQS message handling.

mtkp commented 8 years ago

+1

we should have hooks for both enqueue (similar to clj-http) and dequeue (similar to ring) middlewares

enqueue middleware could, for example, automatically inject some data into SQS message attributes taken from the current threadlocal context.

ghost commented 8 years ago

So I've been working with basically implementing this on the application side to test out one approach, and I think we need the following to add this to the library.

  1. Better document exactly what the Message object coming out of SQS looks like. Right now, it can be found at https://github.com/TheClimateCorporation/squeedo/blob/master/src/com/climate/squeedo/sqs.clj#L162, but it would be nice to not have to dive deep into a private function to find that information, and it can be a pain when you have to do it to properly implement each one of your middlewares.
  2. Add some default middleware as illustrations of how to create your own middleware. Deserialization and logging are obvious choices.
  3. Doing it similar to the way ring works will work well whenever we're consuming from many different queues, as you could define a set of middleware in a single place that will work the same for all queues that you're listening to. However, it is simple enough to wrap your message handler in middleware client side if you're only listening to a single queue. We should add examples of both to the readme.